-----============= acceptance-small: sanityn ============----- Sat Sep 5 23:32:25 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg422-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 23:32:44 (1788665564) === Starting client oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Mount client oleg422-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Started clients oleg422-client.virtnet: 192.168.204.122@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg422-client.virtnet: executing check_config_client /mnt/lustre oleg422-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg422-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9229c84b7800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9229c96ec800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229c84b7800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229c96ec800.idle_timeout=debug disable quota as required oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 23:33:19 (1788665599) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520808 s, 20.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7620] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 23:33:21 (1788665601) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 23:33:30 (1788665610) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 23:33:37 (1788665617) -rw-r--r-- 1 root root 0 Sep 5 23:33 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 23:33:45 (1788665625) -rw-r--r-- 1 root root 0 Sep 5 23:33 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 23:33:52 (1788665632) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (8s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 23:34:00 (1788665640) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (8s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 23:34:08 (1788665648) total 1 -rwxrwxrwx 1 root root 0 Sep 5 23:33 f2a -rwxrwxrwx 1 root root 0 Sep 5 23:33 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 5 23:33 f2a -rwxrwxrwx 1 root root 0 Sep 5 23:33 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0278046 s, 18.4 kB/s PASS 2e (8s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 23:34:16 (1788665656) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 23:34:20 (1788665660) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.506891 s, 4.1 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 23:34:28 (1788665668) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 23:34:34 (1788665674) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 23:34:41 (1788665681) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 23:34:48 (1788665688) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 23:34:56 (1788665696) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 23:35:03 (1788665703) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 23:35:10 (1788665710) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 23:35:18 (1788665718) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676981 s, 15.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0610092 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 23:35:24 (1788665724) 1+0 records in 1+0 records out 1 byte copied, 0.0139829 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127475 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0285368 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128054 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00831955 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00954636 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108117 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132855 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0194113 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.013016 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119189 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0141675 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 23:35:32 (1788665732) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0216131 s, 142 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0716707 s, 57.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000416101 s, 7.4 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 23:35:39 (1788665739) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 23:35:47 (1788665747) warning: '-runtime' deprecated, use '-t runtime' instead - link 274 (time 1788665760.72 total 10.02 last 27.34) - link 502 (time 1788665770.73 total 20.03 last 22.77) - link 765 (time 1788665780.78 total 30.08 last 26.18) - link 1040 (time 1788665790.79 total 40.09 last 27.48) - link 1301 (time 1788665800.79 total 50.09 last 26.09) - link 1634 (time 1788665810.80 total 60.10 last 33.28) - link 1978 (time 1788665820.82 total 70.12 last 34.30) - link 2283 (time 1788665830.82 total 80.12 last 30.50) - link 2568 (time 1788665840.85 total 90.15 last 28.43) total: 2844 link in 100.03 seconds: 28.43 ops/second using seed 2071413497 running for 100 seconds - stat 4161 (time 1788665762 ; total 11 ; last 11) - stat 7996 (time 1788665773 ; total 22 ; last 11) - stat 10000 (time 1788665777 ; total 26 ; last 4) - stat 14077 (time 1788665788 ; total 37 ; last 11) - stat 18887 (time 1788665799 ; total 48 ; last 11) - stat 20000 (time 1788665800 ; total 49 ; last 1) - stat 22560 (time 1788665811 ; total 60 ; last 11) - stat 25206 (time 1788665822 ; total 71 ; last 11) - stat 27560 (time 1788665833 ; total 82 ; last 11) - stat 29739 (time 1788665844 ; total 93 ; last 11) - stat 30000 (time 1788665845 ; total 94 ; last 1) total: 31464 stats in 100 seconds: 314.640015 stats/second - unlinked 0 (time 1788665801 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2844) error: No such file or directory total: 2843 unlinks in 83 seconds: 34.253014 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21418) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21420) - No such process PASS 12 (143s) == sanityn test 13: test directory page revocation ======= 23:38:10 (1788665890) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 23:38:19 (1788665899) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22604 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 23:38:26 (1788665906) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23183 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 23:38:32 (1788665912) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23761 Terminated $DIR1/$tdir/sleep 60 PASS 14b (9s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 23:38:41 (1788665921) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24343 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 23:38:48 (1788665928) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24925 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 23:38:58 (1788665938) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (55s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 23:39:53 (1788665993) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.649239 s, 12.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.34842 s, 24.1 MB/s Chance of close/open is 1 in 50 Seed set to 5999 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x2b0410 000003[1] 1788665999.954918 trunc from 00000000 to 0x2b040f (0x2b0410 bytes) truncating to largest ever: 0x30a101 000005[0] 1788666000.055978 trunc from 0x2b0410 to 0x30a100 (0x59cf1 bytes) 000006[1] 1788666000.156592 mapread 0x1ea988 thru 0x1f9a8c (0xf105 bytes) 000008[0] 1788666000.367802 write 0x0d6bd5 thru 0x0d9fa6 (0x33d2 bytes) 000010[0] 1788666000.465663 write 0x91a0c6 thru 0x91bb2f (0x1a6a bytes) truncating to largest ever: 0x315210 000011[0] 1788666000.470373 trunc from 0x91bb30 to 0x31520f (0x606920 bytes) 000012[1] 1788666000.495669 mapread 0x1b437c thru 0x1b5ff1 (0x1c76 bytes) 000013[0] 1788666000.542082 punch from 0x0c2e2f to 0x0cc228 (0x93fa bytes) 000015[1] 1788666000.595513 mapread 0x27c23e thru 0x28b082 (0xee45 bytes) 000016[0] 1788666000.645916 mapread 0x262fcb thru 0x268eb8 (0x5eee bytes) 000018[0] 1788666000.679095 mapwrite 0x0e94c3 thru 0x0eb5d6 (0x2114 bytes) 000019[0] 1788666000.763429 write 0x62e74f thru 0x63e3a9 (0xfc5b bytes) truncating to largest ever: 0x89403e 000020[0] 1788666000.794390 trunc from 0x63e3aa to 0x89403d (0x255c94 bytes) 000021[1] 1788666000.847134 trunc from 0x89403e to 0x302bae (0x59148f bytes) 000022[1] 1788666000.935586 punch from 0x10a3ac to 0x119165 (0xedba bytes) 000023[0] 1788666000.966532 mapread 0x2127fd thru 0x21db89 (0xb38d bytes) 000024[0] 1788666001.078033 write 0x7e7390 thru 0x7f17a7 (0xa418 bytes) 000025[0] 1788666001.097491 write 0x0cead4 thru 0x0db762 (0xcc8f bytes) 000027[1] 1788666001.135534 punch from 0x34901b to 0x358477 (0xf45d bytes) 000028[1] 1788666001.187141 mapwrite 0x265047 thru 0x26f916 (0xa8d0 bytes) 000029[0] 1788666001.330441 read 0x7059e0 thru 0x71134d (0xb96e bytes) 000032[1] 1788666001.402096 write 0x38b206 thru 0x38f911 (0x470c bytes) 000034[0] 1788666001.434984 trunc from 0x7f17a8 to 0x5b59b2 (0x23bdf5 bytes) 000038[0] 1788666001.554440 punch from 0x40a8c7 to 0x417727 (0xce61 bytes) 000039[1] 1788666001.579584 mapwrite 0x923307 thru 0x9265b7 (0x32b1 bytes) 000040[0] 1788666001.726805 mapwrite 0x9c5642 thru 0x9c601e (0x9dd bytes) 000041[0] 1788666001.839577 mapread 0x52aa4a thru 0x52b96a (0xf21 bytes) 000042[1] 1788666001.877575 read 0x073bfe thru 0x080820 (0xcc23 bytes) 000044[0] 1788666001.924565 punch from 0x92e9a3 to 0x93c582 (0xdbe0 bytes) 000046[0] 1788666002.025118 write 0x1541bc thru 0x155be0 (0x1a25 bytes) 000050[1] 1788666002.166126 read 0x8fb775 thru 0x9036f8 (0x7f84 bytes) 000051[1] 1788666002.219184 read 0x84604e thru 0x855a5e (0xfa11 bytes) 000055[1] 1788666002.294152 punch from 0x337e63 to 0x33bcc0 (0x3e5e bytes) truncating to largest ever: 0x9bbbaf 000056[0] 1788666002.330177 trunc from 0x9c601f to 0x9bbbae (0xa470 bytes) 000057[1] 1788666002.368432 write 0x8e9a10 thru 0x8f5dee (0xc3df bytes) 000059[1] 1788666002.425204 read 0x0a21a9 thru 0x0ae2c2 (0xc11a bytes) 000060[0] 1788666002.453183 mapread 0x233e26 thru 0x23d54d (0x9728 bytes) 000063[1] 1788666002.569097 read 0x7c93bd thru 0x7d5ac5 (0xc709 bytes) 000064[0] 1788666002.598188 write 0x93b2b5 thru 0x942e1b (0x7b67 bytes) 000065[1] 1788666002.644049 mapread 0x088b1e thru 0x08a435 (0x1918 bytes) 000066[1] 1788666002.697223 mapread 0x872ec2 thru 0x876ac4 (0x3c03 bytes) 000068[1] 1788666002.716385 mapread 0x8baa67 thru 0x8c8497 (0xda31 bytes) 000069[1] 1788666002.734001 punch from 0x4cbdb6 to 0x4d6441 (0xa68c bytes) 000070[0] 1788666002.771959 mapread 0x03ed86 thru 0x0479ff (0x8c7a bytes) 000072[1] 1788666002.814326 write 0x67f83a thru 0x686bb0 (0x7377 bytes) 000074[1] 1788666002.861222 mapwrite 0x9f764b thru 0x9fd67d (0x6033 bytes) 000075[1] 1788666003.073909 read 0x34187d thru 0x34278a (0xf0e bytes) 000076[1] 1788666003.089306 trunc from 0x9fd67e to 0x913198 (0xea4e5 bytes) 000077[0] 1788666003.128388 mapread 0x2eb2aa thru 0x2f9db9 (0xeb10 bytes) 000079[1] 1788666003.210542 mapread 0x4af3e9 thru 0x4bd4df (0xe0f7 bytes) 000084[0] 1788666003.386064 read 0x8ea245 thru 0x8f67b7 (0xc573 bytes) 000085[1] 1788666003.406024 read 0x147f69 thru 0x1530e9 (0xb181 bytes) 000087[0] 1788666003.453185 read 0x3e189d thru 0x3f136e (0xfad2 bytes) 000089[0] 1788666003.499057 mapwrite 0x9af557 thru 0x9b7857 (0x8301 bytes) 000090[0] 1788666003.749130 read 0x0e6b06 thru 0x0e9522 (0x2a1d bytes) 000091[1] 1788666003.820610 read 0x5959f3 thru 0x59a1fa (0x4808 bytes) 000092[1] 1788666003.853376 read 0x10766a thru 0x1126d0 (0xb067 bytes) 000093[0] 1788666003.876408 mapread 0x62d184 thru 0x63b21d (0xe09a bytes) 000095[0] 1788666003.911982 mapwrite 0x38fa00 thru 0x392cf7 (0x32f8 bytes) 000097[1] 1788666004.030764 mapread 0x803f35 thru 0x80f7b3 (0xb87f bytes) 000100[1] 1788666004.104439 mapwrite 0x3b2f47 thru 0x3b3144 (0x1fe bytes) 000102[0] 1788666004.239132 mapwrite 0x13e5de thru 0x1461ed (0x7c10 bytes) 000103[0] 1788666004.468304 read 0x68d66a thru 0x69596c (0x8303 bytes) truncating to largest ever: 0x9ced1e 000104[1] 1788666004.492276 trunc from 0x9b7858 to 0x9ced1d (0x174c6 bytes) 000106[1] 1788666004.767895 read 0x0b0896 thru 0x0bdf52 (0xd6bd bytes) 000107[0] 1788666004.796763 mapwrite 0x4bfc9f thru 0x4c344c (0x37ae bytes) 000108[0] 1788666005.169916 punch from 0x782adc to 0x78ec2b (0xc150 bytes) truncating to largest ever: 0x9ff0cc 000109[1] 1788666005.197353 trunc from 0x9ced1e to 0x9ff0cb (0x303ae bytes) 000110[1] 1788666005.282228 read 0x0dccbb thru 0x0de26b (0x15b1 bytes) 000112[0] 1788666005.334703 read 0x1ae3b6 thru 0x1b2a40 (0x468b bytes) 000113[0] 1788666005.365357 mapwrite 0x47b566 thru 0x4843ee (0x8e89 bytes) 000114[0] 1788666005.604691 punch from 0x4d1a16 to 0x4d2fec (0x15d7 bytes) 000116[1] 1788666005.652346 mapwrite 0x493bb8 thru 0x4a0ede (0xd327 bytes) 000117[1] 1788666005.775837 mapwrite 0x719f6d thru 0x721469 (0x74fd bytes) 000119[1] 1788666005.896604 mapread 0x8a7322 thru 0x8a74e6 (0x1c5 bytes) 000120[0] 1788666005.904607 punch from 0x747adb to 0x74b3d1 (0x38f7 bytes) 000121[0] 1788666005.962007 mapread 0x19c44c thru 0x1a2c38 (0x67ed bytes) 000124[0] 1788666005.993983 mapread 0x20640e thru 0x20b347 (0x4f3a bytes) 000126[1] 1788666006.005157 punch from 0x571cdc to 0x57ccfa (0xb01f bytes) 000128[0] 1788666006.048223 punch from 0x1426af to 0x1500b5 (0xda07 bytes) 000131[1] 1788666006.168566 write 0x68eded thru 0x69394d (0x4b61 bytes) 000134[1] 1788666006.243385 mapread 0x67a26c thru 0x68045a (0x61ef bytes) 000135[0] 1788666006.303063 mapwrite 0x2735e9 thru 0x2773a8 (0x3dc0 bytes) 000136[0] 1788666006.398173 punch from 0x5c207b to 0x5c5c54 (0x3bda bytes) 000137[0] 1788666006.428805 read 0x844458 thru 0x84a2af (0x5e58 bytes) 000139[1] 1788666006.494097 mapread 0x3116b6 thru 0x31d4e7 (0xbe32 bytes) 000140[0] 1788666006.569434 punch from 0x7f9108 to 0x7ff3b5 (0x62ae bytes) 000142[0] 1788666006.624565 read 0x4b2de8 thru 0x4bfc2a (0xce43 bytes) 000143[1] 1788666006.686898 trunc from 0x9ff0cc to 0x2aabce (0x7544fd bytes) 000145[0] 1788666006.774310 mapread 0x105af5 thru 0x1077dd (0x1ce9 bytes) 000147[1] 1788666006.866795 read 0x0b750e thru 0x0c4d65 (0xd858 bytes) 000148[0] 1788666006.978988 write 0x49f7cc thru 0x4accf1 (0xd526 bytes) 000149[1] 1788666006.998638 punch from 0x163b9e to 0x16c1da (0x863d bytes) 000150[0] 1788666007.059843 read 0x01109e thru 0x020acd (0xfa30 bytes) 000152[1] 1788666007.118429 mapwrite 0x7e7a08 thru 0x7f1129 (0x9722 bytes) 000155[1] 1788666007.407881 trunc from 0x7f112a to 0x2ecb2b (0x5045fe bytes) 000156[0] 1788666007.464923 mapread 0x0ab4cc thru 0x0b7a7e (0xc5b3 bytes) 000158[1] 1788666007.576872 mapread 0x213c5b thru 0x21e7d7 (0xab7d bytes) 000160[0] 1788666007.718775 punch from 0x2d9b79 to 0x2e3cb9 (0xa141 bytes) 000161[1] 1788666007.757723 mapread 0x2391e7 thru 0x244ad7 (0xb8f1 bytes) 000162[1] 1788666008.038854 write 0x3ae047 thru 0x3b01b2 (0x216c bytes) 000164[0] 1788666008.150514 read 0x2fbc77 thru 0x304c9f (0x9029 bytes) 000167[0] 1788666008.390318 trunc from 0x3b01b3 to 0x5a5815 (0x1f5663 bytes) 000169[1] 1788666008.531162 read 0x0e8397 thru 0x0f0dcc (0x8a36 bytes) 000170[1] 1788666008.671599 mapread 0x2ae1da thru 0x2b2829 (0x4650 bytes) 000174[0] 1788666008.760972 trunc from 0x5a5816 to 0x86e6bb (0x2c8ea6 bytes) 000175[1] 1788666008.839721 read 0x303c4b thru 0x3139e6 (0xfd9c bytes) 000177[1] 1788666008.912605 read 0x66e840 thru 0x672c24 (0x43e5 bytes) 000178[0] 1788666008.967282 trunc from 0x86e6bc to 0x4fa134 (0x374587 bytes) 000179[1] 1788666009.081859 mapread 0x0915cc thru 0x09f4dd (0xdf12 bytes) 000180[0] 1788666009.149684 trunc from 0x4fa135 to 0x028fef (0x4d1145 bytes) 000181[0] 1788666009.190130 punch from 0x027463 to 0x028fef (0x1b8d bytes) 000183[0] 1788666009.199043 read 0x011d02 thru 0x01c734 (0xaa33 bytes) 000184[0] 1788666009.238708 mapread 0x003df2 thru 0x0093d2 (0x55e1 bytes) 000186[0] 1788666009.289184 read 0x01af7b thru 0x01e9ee (0x3a74 bytes) 000187[0] 1788666009.317138 mapread 0x0203ba thru 0x026d63 (0x69aa bytes) 000192[1] 1788666009.434137 mapread 0x00286d thru 0x004cb9 (0x244d bytes) 000193[1] 1788666009.471582 read 0x019e58 thru 0x024bdb (0xad84 bytes) 000194[1] 1788666009.483794 trunc from 0x028ff0 to 0x1b4a9c (0x18baad bytes) 000195[1] 1788666009.582971 read 0x00ccb8 thru 0x019ce0 (0xd029 bytes) 000198[1] 1788666009.643644 mapread 0x17cc4c thru 0x189634 (0xc9e9 bytes) 000199[1] 1788666009.693306 punch from 0x058dd5 to 0x05f6fc (0x6928 bytes) 000200[1] 1788666009.729204 punch from 0x0d73ee to 0x0dd9c3 (0x65d6 bytes) 000203[1] 1788666009.777769 read 0x094867 thru 0x0a279e (0xdf38 bytes) 000204[1] 1788666009.780817 trunc from 0x1b4a9d to 0x65c3ea (0x4a794e bytes) 000206[0] 1788666009.847728 trunc from 0x65c3eb to 0x8f8511 (0x29c127 bytes) 000207[1] 1788666009.913967 trunc from 0x8f8512 to 0x424be7 (0x4d392a bytes) 000208[0] 1788666009.967171 read 0x2709ca thru 0x27fd86 (0xf3bd bytes) 000210[1] 1788666010.022556 trunc from 0x424be8 to 0x59e6f0 (0x179b09 bytes) 000212[0] 1788666010.208280 punch from 0x150caf to 0x156857 (0x5ba9 bytes) 000213[1] 1788666010.290528 mapread 0x0fd1ab thru 0x10acb7 (0xdb0d bytes) 000215[1] 1788666010.436336 write 0x0c62c0 thru 0x0cf3d3 (0x9114 bytes) 000216[1] 1788666010.501037 mapread 0x0e34a9 thru 0x0ead34 (0x788c bytes) 000217[0] 1788666010.577109 punch from 0x5656ee to 0x5709e8 (0xb2fb bytes) 000218[0] 1788666010.618212 mapread 0x4ac7e2 thru 0x4ae4a9 (0x1cc8 bytes) 000219[1] 1788666010.693934 read 0x2e40fb thru 0x2e75ac (0x34b2 bytes) 000220[0] 1788666010.717690 punch from 0x000468 to 0x0070ac (0x6c45 bytes) 000221[1] 1788666010.766216 write 0x568eba thru 0x573db9 (0xaf00 bytes) 000223[0] 1788666010.868180 mapread 0x431c1d thru 0x43e20e (0xc5f2 bytes) 000224[0] 1788666011.023740 mapread 0x36146f thru 0x363759 (0x22eb bytes) 000225[1] 1788666011.078968 punch from 0x4e2270 to 0x4e4965 (0x26f6 bytes) 000226[0] 1788666011.117685 read 0x361109 thru 0x36299b (0x1893 bytes) 000228[1] 1788666011.152513 trunc from 0x59e6f1 to 0x7ddd4e (0x23f65e bytes) 000229[0] 1788666011.206966 trunc from 0x7ddd4f to 0x6fcdb2 (0xe0f9c bytes) 000230[0] 1788666011.258101 mapwrite 0x58998e thru 0x590581 (0x6bf4 bytes) 000233[0] 1788666011.337309 mapread 0x526a0e thru 0x530d51 (0xa344 bytes) 000234[1] 1788666011.425996 mapwrite 0x7e81a9 thru 0x7e93d2 (0x122a bytes) 000237[1] 1788666011.730872 punch from 0x3d3bac to 0x3d8286 (0x46db bytes) 000238[1] 1788666011.739694 read 0x6ba210 thru 0x6bc2a5 (0x2096 bytes) 000240[1] 1788666011.771178 read 0x6ef837 thru 0x6fbe24 (0xc5ee bytes) 000242[1] 1788666011.815871 mapread 0x37d1ab thru 0x38bcc8 (0xeb1e bytes) 000244[1] 1788666011.878771 read 0x30abda thru 0x316cb8 (0xc0df bytes) 000245[1] 1788666011.892596 punch from 0x461e57 to 0x4686a4 (0x684e bytes) 000246[0] 1788666011.923840 mapwrite 0x36fc76 thru 0x373c3a (0x3fc5 bytes) 000248[0] 1788666012.126041 punch from 0x460e87 to 0x4652e3 (0x445d bytes) 000249[0] 1788666012.186262 read 0x092e65 thru 0x096055 (0x31f1 bytes) 000251[0] 1788666012.255833 write 0x4543aa thru 0x459000 (0x4c57 bytes) 000252[1] 1788666012.272597 punch from 0x0e8c53 to 0x0ecad8 (0x3e86 bytes) 000253[1] 1788666012.352646 punch from 0x05045d to 0x05b9a8 (0xb54c bytes) 000254[1] 1788666012.392225 mapwrite 0x84fa9f thru 0x859d78 (0xa2da bytes) 000256[1] 1788666012.539446 punch from 0x6718d9 to 0x678389 (0x6ab1 bytes) 000257[1] 1788666012.599206 mapwrite 0x73e426 thru 0x741722 (0x32fd bytes) 000258[0] 1788666012.765735 mapwrite 0x04679d thru 0x050329 (0x9b8d bytes) 000260[1] 1788666012.940099 mapread 0x6e9485 thru 0x6eb062 (0x1bde bytes) 000264[0] 1788666013.119912 punch from 0x382c49 to 0x38e51e (0xb8d6 bytes) 000265[0] 1788666013.165965 mapwrite 0x184d69 thru 0x1934a1 (0xe739 bytes) 000266[1] 1788666013.436867 write 0x2946be thru 0x29fc11 (0xb554 bytes) 000267[0] 1788666013.497756 punch from 0x3faca1 to 0x407bc0 (0xcf20 bytes) 000269[0] 1788666013.600903 mapwrite 0x660c52 thru 0x667433 (0x67e2 bytes) 000270[1] 1788666013.747411 mapwrite 0x966e39 thru 0x96cc86 (0x5e4e bytes) 000271[1] 1788666013.977214 write 0x5557aa thru 0x563bb6 (0xe40d bytes) 000272[0] 1788666013.995171 punch from 0x5d104b to 0x5dcd5a (0xbd10 bytes) 000273[0] 1788666014.066922 write 0x3b628a thru 0x3bceec (0x6c63 bytes) 000274[1] 1788666014.092866 read 0x08e3f8 thru 0x096a78 (0x8681 bytes) 000275[1] 1788666014.127975 mapread 0x8fa6d1 thru 0x904bc7 (0xa4f7 bytes) 000277[1] 1788666014.185780 mapread 0x90786d thru 0x91301f (0xb7b3 bytes) 000278[0] 1788666014.224958 mapwrite 0x224a26 thru 0x22a162 (0x573d bytes) 000279[1] 1788666014.310228 mapwrite 0x44ffad thru 0x4590c9 (0x911d bytes) 000284[1] 1788666014.605597 mapread 0x95aa3d thru 0x968848 (0xde0c bytes) 000285[1] 1788666014.783763 punch from 0x2bfb8e to 0x2c81ad (0x8620 bytes) 000286[0] 1788666014.860084 mapwrite 0x182f96 thru 0x184728 (0x1793 bytes) 000288[0] 1788666015.091679 mapwrite 0x7a7955 thru 0x7aa1ac (0x2858 bytes) 000289[0] 1788666015.299428 mapread 0x52c3ed thru 0x5304e4 (0x40f8 bytes) 000290[1] 1788666015.322654 trunc from 0x96cc87 to 0x0ccbf8 (0x8a008e bytes) 000293[0] 1788666015.430592 mapwrite 0x4278d5 thru 0x431bc9 (0xa2f5 bytes) 000294[0] 1788666015.676148 mapread 0x29734c thru 0x29a63a (0x32ef bytes) 000295[1] 1788666015.719575 read 0x3c5fd3 thru 0x3ce3a7 (0x83d5 bytes) 000296[0] 1788666015.774796 punch from 0x092d7b to 0x0993ae (0x6634 bytes) 000299[0] 1788666015.908955 read 0x33f30e thru 0x3464f0 (0x71e3 bytes) 000301[1] 1788666015.979157 write 0x771c1f thru 0x77c4e0 (0xa8c2 bytes) 000302[1] 1788666016.037547 trunc from 0x77c4e1 to 0x526227 (0x2562b9 bytes) 000304[1] 1788666016.263055 write 0x8ddf83 thru 0x8e3ef3 (0x5f71 bytes) 000305[1] 1788666016.311645 mapread 0x84658c thru 0x8471b4 (0xc29 bytes) 000307[1] 1788666016.398300 punch from 0x226241 to 0x229194 (0x2f54 bytes) 000309[1] 1788666016.504792 mapread 0x187806 thru 0x19203c (0xa837 bytes) 000310[1] 1788666016.625902 mapwrite 0x8cecf1 thru 0x8d4eb2 (0x61c2 bytes) 000311[0] 1788666016.695195 write 0x0520b1 thru 0x058b34 (0x6a84 bytes) 000312[0] 1788666016.732728 mapwrite 0x59336a thru 0x593587 (0x21e bytes) 000314[0] 1788666016.910455 trunc from 0x8e3ef4 to 0x4339f2 (0x4b0501 bytes) 000316[0] 1788666016.943610 mapread 0x2a65ba thru 0x2aa188 (0x3bcf bytes) 000317[1] 1788666017.049846 mapwrite 0x52cfe9 thru 0x5352b4 (0x82cc bytes) 000318[0] 1788666017.308196 write 0x2bd5c7 thru 0x2cc4f1 (0xef2b bytes) 000319[1] 1788666017.357414 punch from 0x2e0989 to 0x2e290c (0x1f84 bytes) 000320[1] 1788666017.390266 write 0x95a9b2 thru 0x964b23 (0xa172 bytes) 000322[0] 1788666017.432423 mapread 0x80058a thru 0x80e173 (0xdbea bytes) 000324[1] 1788666017.656673 punch from 0x3af688 to 0x3b449b (0x4e14 bytes) 000325[1] 1788666017.741788 trunc from 0x964b24 to 0x4c6061 (0x49eac2 bytes) 000328[0] 1788666017.796065 punch from 0x10ee7b to 0x11a058 (0xb1de bytes) 000329[1] 1788666017.827447 mapwrite 0x927ab1 thru 0x9318b7 (0x9e07 bytes) 000330[0] 1788666018.051705 mapwrite 0x91bd69 thru 0x922e56 (0x70ee bytes) 000331[0] 1788666018.165824 trunc from 0x9318b8 to 0x2cc31f (0x665598 bytes) 000332[1] 1788666018.204103 mapread 0x0ab254 thru 0x0ab7c9 (0x576 bytes) 000335[1] 1788666018.280734 write 0x1e8e51 thru 0x1f7097 (0xe247 bytes) 000336[0] 1788666018.323233 read 0x210ed0 thru 0x21bbf6 (0xad27 bytes) 000337[0] 1788666018.471540 mapread 0x17bbda thru 0x17d101 (0x1528 bytes) 000339[1] 1788666018.501029 punch from 0x004b3a to 0x00be61 (0x7328 bytes) 000345[1] 1788666018.719971 mapwrite 0x60aacc thru 0x60e7e0 (0x3d15 bytes) 000346[0] 1788666018.975636 trunc from 0x60e7e1 to 0x7c133f (0x1b2b5f bytes) 000347[1] 1788666019.103953 read 0x7793e4 thru 0x7859a1 (0xc5be bytes) 000348[1] 1788666019.170113 mapwrite 0x35c735 thru 0x35deb0 (0x177c bytes) 000349[0] 1788666019.361912 read 0x0d60da thru 0x0e3d9e (0xdcc5 bytes) 000350[1] 1788666019.408181 write 0x20553e thru 0x20e7ed (0x92b0 bytes) 000351[0] 1788666019.466977 punch from 0x18a15f to 0x199f0e (0xfdb0 bytes) 000352[0] 1788666019.524155 punch from 0x21a29a to 0x222fff (0x8d66 bytes) 000353[1] 1788666019.571087 mapread 0x0d64f8 thru 0x0dd612 (0x711b bytes) 000357[1] 1788666019.636899 punch from 0x5b80ab to 0x5bf760 (0x76b6 bytes) 000360[1] 1788666019.727895 punch from 0x657196 to 0x65bf6e (0x4dd9 bytes) 000361[0] 1788666019.735585 write 0x606bf6 thru 0x60eb74 (0x7f7f bytes) 000362[1] 1788666019.830417 write 0x1ac7e6 thru 0x1b77db (0xaff6 bytes) 000364[1] 1788666019.966898 trunc from 0x7c1340 to 0x37cf21 (0x44441e bytes) 000366[0] 1788666020.124529 read 0x07ecfe thru 0x08d7f1 (0xeaf4 bytes) 000368[1] 1788666020.303627 trunc from 0x37cf22 to 0x1759fd (0x207524 bytes) 000370[0] 1788666020.347599 mapwrite 0x011938 thru 0x02031a (0xe9e3 bytes) 000371[0] 1788666020.617211 mapread 0x0eaa68 thru 0x0ef7eb (0x4d84 bytes) 000372[0] 1788666020.738042 mapread 0x05eb57 thru 0x060cb1 (0x215b bytes) 000373[0] 1788666020.797259 trunc from 0x1759fe to 0x75fc4c (0x5ea24f bytes) 000375[1] 1788666020.914734 punch from 0x1f1274 to 0x1fb9db (0xa768 bytes) 000376[0] 1788666020.955517 punch from 0x1a7cd8 to 0x1ada5c (0x5d85 bytes) 000377[0] 1788666020.976862 punch from 0x1f963a to 0x1fefe2 (0x59a9 bytes) 000378[1] 1788666020.982745 punch from 0x1d61cc to 0x1daaf9 (0x492e bytes) 000379[0] 1788666021.023873 trunc from 0x75fc4d to 0x9f546f (0x295823 bytes) 000380[1] 1788666021.089854 write 0x00d9bc thru 0x0157f0 (0x7e35 bytes) 000381[0] 1788666021.142011 write 0x834e34 thru 0x83c9b6 (0x7b83 bytes) 000384[0] 1788666021.500211 mapwrite 0x3563f2 thru 0x35aab0 (0x46bf bytes) 000385[1] 1788666021.625074 trunc from 0x9f5470 to 0x189611 (0x86be5e bytes) 000387[1] 1788666021.730001 mapwrite 0x33c346 thru 0x33deef (0x1baa bytes) 000392[0] 1788666021.870128 write 0x395237 thru 0x3a1939 (0xc703 bytes) 000394[1] 1788666021.925830 mapread 0x31ff4a thru 0x32f97d (0xfa34 bytes) 000397[1] 1788666022.020884 write 0x977b10 thru 0x983692 (0xbb83 bytes) 000403[1] 1788666022.212874 read 0x48f520 thru 0x492969 (0x344a bytes) 000406[1] 1788666022.298430 write 0x95ef23 thru 0x96e128 (0xf206 bytes) 000408[1] 1788666022.393741 mapwrite 0x39cb7c thru 0x3abfc5 (0xf44a bytes) 000410[0] 1788666022.615820 trunc from 0x983693 to 0x049e52 (0x939840 bytes) 000411[1] 1788666022.724965 mapread 0x00d9e9 thru 0x01ceb5 (0xf4cd bytes) 000414[0] 1788666022.913363 punch from 0x01e05f to 0x024e09 (0x6dab bytes) 000415[1] 1788666023.050977 punch from 0x047be1 to 0x049e52 (0x2272 bytes) 000416[1] 1788666023.168332 read 0x028a83 thru 0x029e0e (0x138c bytes) 000417[0] 1788666023.219841 mapread 0x01dbee thru 0x0204b6 (0x28c9 bytes) 000419[1] 1788666023.291518 mapwrite 0x5090cc thru 0x51829a (0xf1cf bytes) 000420[0] 1788666023.597316 trunc from 0x51829b to 0x3386f8 (0x1dfba2 bytes) 000421[1] 1788666023.712901 mapread 0x2481be thru 0x2539d1 (0xb814 bytes) 000423[0] 1788666023.830510 punch from 0x005868 to 0x01184f (0xbfe8 bytes) 000424[1] 1788666023.878187 trunc from 0x3386f9 to 0x84d780 (0x515088 bytes) 000425[0] 1788666023.965823 mapread 0x18ac30 thru 0x1992ca (0xe69b bytes) 000426[0] 1788666024.139085 mapwrite 0x0ee469 thru 0x0f39eb (0x5583 bytes) 000429[1] 1788666024.349613 read 0x712475 thru 0x720291 (0xde1d bytes) 000430[1] 1788666024.378330 punch from 0x4cbb73 to 0x4dba5e (0xfeec bytes) 000431[0] 1788666024.396688 trunc from 0x84d781 to 0x2bd29e (0x5904e2 bytes) 000432[1] 1788666024.455929 mapwrite 0x1bcfec thru 0x1c6edd (0x9ef2 bytes) 000433[1] 1788666024.675814 mapread 0x0b660d thru 0x0b9c95 (0x3689 bytes) 000434[1] 1788666024.781638 punch from 0x1002cc to 0x106c18 (0x694d bytes) 000436[0] 1788666024.853153 mapwrite 0x51ccf8 thru 0x52c215 (0xf51e bytes) 000438[0] 1788666025.171042 write 0x59e2ea thru 0x5ac9fe (0xe715 bytes) 000443[1] 1788666025.292662 mapwrite 0x5a03f1 thru 0x5aace9 (0xa8f9 bytes) 000444[0] 1788666025.432040 mapwrite 0x8eeef9 thru 0x8f2aff (0x3c07 bytes) 000445[1] 1788666025.568048 punch from 0x53c3df to 0x547018 (0xac3a bytes) 000446[1] 1788666025.637913 read 0x77ea03 thru 0x78d0c4 (0xe6c2 bytes) 000448[1] 1788666025.729899 mapwrite 0x9401fb thru 0x94b99d (0xb7a3 bytes) 000449[1] 1788666025.865951 trunc from 0x94b99e to 0x40b56d (0x540430 bytes) 000451[0] 1788666025.892965 write 0x67621d thru 0x684303 (0xe0e7 bytes) 000454[1] 1788666026.002094 punch from 0x03c4fb to 0x049bd3 (0xd6d9 bytes) 000455[0] 1788666026.028315 mapwrite 0x3f2e10 thru 0x3f53d5 (0x25c6 bytes) 000456[1] 1788666026.197958 mapread 0x3ebc0f thru 0x3f9493 (0xd885 bytes) 000457[0] 1788666026.272650 mapwrite 0x720267 thru 0x72b306 (0xb0a0 bytes) 000459[0] 1788666026.526867 punch from 0x141b81 to 0x145992 (0x3e12 bytes) 000460[0] 1788666026.553612 write 0x596e79 thru 0x59af25 (0x40ad bytes) 000462[0] 1788666026.701820 mapread 0x476eb0 thru 0x47857d (0x16ce bytes) 000463[1] 1788666026.765684 mapread 0x0043fb thru 0x004844 (0x44a bytes) 000464[0] 1788666026.826069 punch from 0x68e827 to 0x69968e (0xae68 bytes) 000465[1] 1788666026.851276 punch from 0x1f3629 to 0x1f5880 (0x2258 bytes) 000466[1] 1788666026.926440 write 0x47be41 thru 0x4814cd (0x568d bytes) 000471[1] 1788666027.156454 mapwrite 0x6efa71 thru 0x6f2839 (0x2dc9 bytes) 000474[0] 1788666027.387564 mapread 0x536bf2 thru 0x542b9e (0xbfad bytes) 000478[1] 1788666027.570922 write 0x389f97 thru 0x39875f (0xe7c9 bytes) 000480[0] 1788666027.646174 mapread 0x5a6373 thru 0x5acf26 (0x6bb4 bytes) 000481[0] 1788666027.774446 trunc from 0x72b307 to 0x577069 (0x1b429d bytes) 000482[1] 1788666027.848080 read 0x01b36e thru 0x023f76 (0x8c09 bytes) 000483[1] 1788666027.872595 trunc from 0x57706a to 0x1ec75b (0x38a90e bytes) 000484[0] 1788666027.939201 write 0x038abf thru 0x03f21d (0x675f bytes) 000487[1] 1788666028.072024 punch from 0x05de41 to 0x061d00 (0x3ec0 bytes) 000489[1] 1788666028.173543 punch from 0x0a6202 to 0x0b5a6e (0xf86d bytes) 000492[0] 1788666028.231719 trunc from 0x1ec75c to 0x03fd4a (0x1aca11 bytes) 000493[0] 1788666028.271185 mapwrite 0x47f448 thru 0x48871b (0x92d4 bytes) 000494[1] 1788666028.352050 mapread 0x023b6f thru 0x027301 (0x3793 bytes) 000495[0] 1788666028.416825 mapwrite 0x318930 thru 0x320850 (0x7f21 bytes) 000496[0] 1788666028.491888 mapread 0x14940a thru 0x14f8b1 (0x64a8 bytes) 000497[1] 1788666028.510517 write 0x04f35c thru 0x04ff81 (0xc26 bytes) 000498[0] 1788666028.543995 punch from 0x029602 to 0x02e3c4 (0x4dc3 bytes) 000499[1] 1788666028.592838 write 0x02da52 thru 0x034de0 (0x738f bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6028 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x24769c 000003[1] 1788666029.583000 trunc from 00000000 to 0x24769b (0x24769c bytes) 000004[1] 1788666029.627819 punch from 0x005869 to 0x011cb3 (0xc44b bytes) 000006[1] 1788666029.661156 trunc from 0x24769c to 0x20859e (0x3f0fd bytes) 000007[1] 1788666029.696322 mapwrite 0x929000 thru 0x930020 (0x7021 bytes) 000008[0] 1788666029.893536 read 0x01c000 thru 0x02161e (0x561f bytes) 000009[0] 1788666029.932025 mapread 0x8fb000 thru 0x8fbd6a (0xd6b bytes) 000010[1] 1788666029.951664 read 0x78a000 thru 0x78fa79 (0x5a7a bytes) 000012[0] 1788666029.969010 punch from 0x7c3840 to 0x7d275f (0xef20 bytes) 000014[1] 1788666030.074105 mapwrite 0x80b000 thru 0x81837f (0xd380 bytes) 000015[1] 1788666030.239577 mapwrite 0x8ed000 thru 0x8fa204 (0xd205 bytes) 000019[1] 1788666030.357967 punch from 0x1f3828 to 0x1fea43 (0xb21c bytes) 000020[0] 1788666030.419840 write 0x413000 thru 0x422298 (0xf299 bytes) truncating to largest ever: 0x5708f7 000023[0] 1788666030.489645 trunc from 0x930021 to 0x5708f6 (0x3bf72a bytes) 000024[1] 1788666030.546683 punch from 0x292519 to 0x29891b (0x6403 bytes) 000029[1] 1788666030.655000 trunc from 0x5708f7 to 0x0c695d (0x4a9f99 bytes) 000033[0] 1788666030.817438 mapread 0x028000 thru 0x02f423 (0x7424 bytes) 000034[0] 1788666030.891442 punch from 0x05848f to 0x067a20 (0xf592 bytes) 000035[1] 1788666030.918798 mapwrite 0x002000 thru 0x0050d4 (0x30d5 bytes) 000036[0] 1788666031.094160 punch from 0x084a3f to 0x089b4d (0x510f bytes) 000037[0] 1788666031.172527 write 0x3b8000 thru 0x3c0642 (0x8643 bytes) 000039[0] 1788666031.185415 mapread 0x0fc000 thru 0x108c1b (0xcc1c bytes) 000040[0] 1788666031.310982 trunc from 0x3c0643 to 0x329cdb (0x96967 bytes) 000041[1] 1788666031.344141 punch from 0x2fbbe5 to 0x2ff6d4 (0x3af0 bytes) 000042[1] 1788666031.366991 read 0x184000 thru 0x18b4aa (0x74ab bytes) 000044[0] 1788666031.474598 read 0x136000 thru 0x1450e8 (0xf0e9 bytes) 000045[0] 1788666031.598634 mapwrite 0x799000 thru 0x79c115 (0x3116 bytes) 000046[0] 1788666031.889959 punch from 0x6821a1 to 0x682c48 (0xaa8 bytes) 000047[1] 1788666031.915734 write 0x291000 thru 0x29c7d4 (0xb7d5 bytes) truncating to largest ever: 0x7ac0ef 000048[1] 1788666032.044844 trunc from 0x79c116 to 0x7ac0ee (0xffd9 bytes) 000049[1] 1788666032.126780 trunc from 0x7ac0ef to 0x068c93 (0x74345b bytes) 000050[0] 1788666032.176045 mapread 0x026000 thru 0x02d1d7 (0x71d8 bytes) 000052[1] 1788666032.320527 read 0x047000 thru 0x047de5 (0xde6 bytes) 000053[1] 1788666032.348777 mapread 0x057000 thru 0x0653d2 (0xe3d3 bytes) 000054[0] 1788666032.350691 write 0x286000 thru 0x28633b (0x33c bytes) 000056[1] 1788666032.407199 mapread 0x14e000 thru 0x15195e (0x395f bytes) 000057[1] 1788666032.493316 write 0x18d000 thru 0x190f21 (0x3f22 bytes) 000058[0] 1788666032.558427 punch from 0x0bbd70 to 0x0c76fd (0xb98e bytes) 000060[1] 1788666032.612824 write 0x569000 thru 0x576e0c (0xde0d bytes) 000062[0] 1788666032.657584 write 0x322000 thru 0x330468 (0xe469 bytes) 000063[0] 1788666032.721414 trunc from 0x576e0d to 0x567c67 (0xf1a5 bytes) 000064[1] 1788666032.755859 write 0x938000 thru 0x93b001 (0x3002 bytes) 000065[1] 1788666032.801553 punch from 0x2bb686 to 0x2bda44 (0x23bf bytes) 000066[0] 1788666032.811627 write 0x181000 thru 0x182fee (0x1fef bytes) 000067[0] 1788666032.867566 punch from 0x5eda33 to 0x5f57a8 (0x7d76 bytes) 000068[1] 1788666032.873630 write 0x279000 thru 0x28538a (0xc38b bytes) 000070[1] 1788666032.993556 mapwrite 0x4db000 thru 0x4df65c (0x465d bytes) 000072[0] 1788666033.195219 punch from 0x768fb2 to 0x76c64b (0x369a bytes) 000073[0] 1788666033.257955 read 0x134000 thru 0x13e5c8 (0xa5c9 bytes) 000074[0] 1788666033.312622 punch from 0x186f7e to 0x18ec4d (0x7cd0 bytes) 000077[0] 1788666033.395114 write 0x20a000 thru 0x213ee3 (0x9ee4 bytes) 000078[1] 1788666033.403939 read 0x1cf000 thru 0x1decfa (0xfcfb bytes) 000079[1] 1788666033.459416 read 0x156000 thru 0x157be0 (0x1be1 bytes) 000080[0] 1788666033.468668 mapwrite 0x630000 thru 0x63bb3e (0xbb3f bytes) 000083[0] 1788666033.606770 write 0x811000 thru 0x81d66c (0xc66d bytes) 000084[0] 1788666033.650842 write 0x1fb000 thru 0x2084ca (0xd4cb bytes) 000085[0] 1788666033.691887 punch from 0x81ea49 to 0x82b59b (0xcb53 bytes) 000090[0] 1788666033.737197 mapread 0x203000 thru 0x204ff8 (0x1ff9 bytes) 000092[1] 1788666033.755312 mapread 0x3a8000 thru 0x3b4b9b (0xcb9c bytes) 000094[0] 1788666033.821144 write 0x68c000 thru 0x6943f3 (0x83f4 bytes) 000096[1] 1788666033.864599 write 0x180000 thru 0x188b6b (0x8b6c bytes) 000098[0] 1788666033.896128 punch from 0x7f4dad to 0x7fe200 (0x9454 bytes) 000100[1] 1788666033.950931 mapwrite 0x52b000 thru 0x535e22 (0xae23 bytes) 000101[1] 1788666034.063927 mapread 0x6eb000 thru 0x6eb26e (0x26f bytes) 000102[0] 1788666034.080322 mapwrite 0x1f2000 thru 0x1f52d5 (0x32d6 bytes) 000103[1] 1788666034.222838 mapwrite 0x066000 thru 0x06fbf7 (0x9bf8 bytes) 000104[1] 1788666034.350169 mapread 0x11b000 thru 0x11e479 (0x347a bytes) 000105[0] 1788666034.369180 write 0x258000 thru 0x2673bf (0xf3c0 bytes) 000106[0] 1788666034.416150 punch from 0x7b3359 to 0x7ba0f9 (0x6da1 bytes) 000107[0] 1788666034.437200 trunc from 0x93b002 to 0x3a27eb (0x598816 bytes) 000109[1] 1788666034.481359 mapwrite 0x06b000 thru 0x074052 (0x9053 bytes) 000111[0] 1788666034.602332 punch from 0x35a485 to 0x368abb (0xe637 bytes) 000112[1] 1788666034.618819 punch from 0x365b26 to 0x371726 (0xbc01 bytes) 000113[1] 1788666034.651300 punch from 0x280ca2 to 0x28373a (0x2a99 bytes) 000114[0] 1788666034.656723 read 0x11d000 thru 0x122280 (0x5281 bytes) 000115[0] 1788666034.743217 write 0x7ab000 thru 0x7b1614 (0x6615 bytes) 000117[1] 1788666034.769210 read 0x590000 thru 0x59e0a7 (0xe0a8 bytes) 000119[0] 1788666034.917877 trunc from 0x7b1615 to 0x54e4e1 (0x263133 bytes) 000120[0] 1788666034.971863 trunc from 0x54e4e2 to 0x670433 (0x121f52 bytes) 000121[0] 1788666034.990439 write 0x174000 thru 0x180cff (0xcd00 bytes) 000123[0] 1788666035.041840 write 0x041000 thru 0x04f67f (0xe680 bytes) 000124[1] 1788666035.048599 punch from 0x0abd0c to 0x0b9157 (0xd44c bytes) 000126[0] 1788666035.093741 mapread 0x11d000 thru 0x11d809 (0x80a bytes) 000129[0] 1788666035.129609 write 0x07f000 thru 0x080083 (0x1084 bytes) 000131[1] 1788666035.161700 read 0x16a000 thru 0x175880 (0xb881 bytes) 000133[1] 1788666035.204863 mapwrite 0x3ac000 thru 0x3b0047 (0x4048 bytes) 000135[1] 1788666035.288302 punch from 0x2dac3f to 0x2e922e (0xe5f0 bytes) 000136[0] 1788666035.308374 mapread 0x322000 thru 0x32e6f8 (0xc6f9 bytes) 000140[1] 1788666035.405764 trunc from 0x670434 to 0x67c8ac (0xc479 bytes) 000144[0] 1788666035.538157 punch from 0x57db5c to 0x57f1cd (0x1672 bytes) 000145[1] 1788666035.609971 read 0x503000 thru 0x511d30 (0xed31 bytes) 000146[1] 1788666035.657990 write 0x7a4000 thru 0x7a40b0 (0xb1 bytes) 000147[1] 1788666035.718236 write 0x46e000 thru 0x47c825 (0xe826 bytes) 000149[0] 1788666035.778149 mapread 0x064000 thru 0x065b75 (0x1b76 bytes) 000151[1] 1788666035.840844 mapwrite 0x0ba000 thru 0x0be4f0 (0x44f1 bytes) 000153[1] 1788666035.982854 punch from 0x2fa149 to 0x30415e (0xa016 bytes) 000155[0] 1788666036.003706 mapread 0x260000 thru 0x260e29 (0xe2a bytes) 000156[1] 1788666036.034072 read 0x04c000 thru 0x053626 (0x7627 bytes) 000157[1] 1788666036.052918 mapwrite 0x11f000 thru 0x1279cf (0x89d0 bytes) 000158[0] 1788666036.147321 trunc from 0x7a40b1 to 0x1e4a29 (0x5bf687 bytes) truncating to largest ever: 0x7fc3de 000159[1] 1788666036.305660 trunc from 0x1e4a2a to 0x7fc3dd (0x6179b4 bytes) 000161[1] 1788666036.407092 mapread 0x66c000 thru 0x67627a (0xa27b bytes) 000162[1] 1788666036.450373 trunc from 0x7fc3de to 0x18ae97 (0x671546 bytes) 000163[0] 1788666036.482219 read 0x0e6000 thru 0x0f0764 (0xa765 bytes) 000165[0] 1788666036.518813 punch from 0x009a42 to 0x0171be (0xd77d bytes) 000166[1] 1788666036.539843 write 0x92a000 thru 0x930400 (0x6401 bytes) 000167[1] 1788666036.597805 trunc from 0x930401 to 0x3eba19 (0x5449e7 bytes) 000168[0] 1788666036.630671 mapread 0x16f000 thru 0x173378 (0x4379 bytes) 000169[1] 1788666036.699493 mapwrite 0x252000 thru 0x25eb6b (0xcb6c bytes) 000170[1] 1788666036.904744 mapread 0x12a000 thru 0x130e4b (0x6e4c bytes) 000171[1] 1788666036.992388 trunc from 0x3eba1a to 0x141de6 (0x2a9c33 bytes) 000173[1] 1788666037.068734 read 0x08a000 thru 0x097eb3 (0xdeb4 bytes) 000175[1] 1788666037.137498 mapwrite 0x173000 thru 0x17edd5 (0xbdd6 bytes) 000176[1] 1788666037.303880 punch from 0x103291 to 0x10e427 (0xb197 bytes) 000177[1] 1788666037.352441 write 0x68b000 thru 0x697414 (0xc415 bytes) 000178[0] 1788666037.359510 write 0x910000 thru 0x917767 (0x7768 bytes) 000179[0] 1788666037.400249 read 0x378000 thru 0x37dcc9 (0x5cca bytes) truncating to largest ever: 0x8fd8a4 000181[1] 1788666037.426728 trunc from 0x917768 to 0x8fd8a3 (0x19ec4 bytes) 000182[1] 1788666037.499635 mapread 0x674000 thru 0x681e0e (0xde0f bytes) 000183[1] 1788666037.648830 mapread 0x5ee000 thru 0x5ee466 (0x467 bytes) 000187[1] 1788666037.680994 read 0x3b1000 thru 0x3bc99b (0xb99c bytes) 000188[1] 1788666037.711749 mapread 0x006000 thru 0x00a689 (0x468a bytes) 000190[0] 1788666037.766802 mapwrite 0x136000 thru 0x13d22f (0x7230 bytes) 000192[1] 1788666037.915780 mapread 0x1c1000 thru 0x1c23db (0x13dc bytes) 000196[1] 1788666037.968155 read 0x433000 thru 0x439979 (0x697a bytes) 000197[1] 1788666037.980841 trunc from 0x8fd8a4 to 0x080b00 (0x87cda3 bytes) 000200[0] 1788666038.072209 read 0x013000 thru 0x014525 (0x1526 bytes) 000201[1] 1788666038.122842 mapread 0x035000 thru 0x03af4e (0x5f4f bytes) 000206[0] 1788666038.194406 mapread 0x039000 thru 0x03fcc2 (0x6cc3 bytes) 000209[1] 1788666038.196320 read 0x016000 thru 0x024041 (0xe042 bytes) 000210[0] 1788666038.214076 punch from 0x012a76 to 0x01d10a (0xa695 bytes) 000211[0] 1788666038.248287 punch from 0x0269de to 0x02f8f3 (0x8f16 bytes) 000217[0] 1788666038.361698 mapwrite 0x2c5000 thru 0x2cd44c (0x844d bytes) 000218[1] 1788666038.500781 trunc from 0x2cd44d to 0x84e72e (0x5812e2 bytes) 000219[0] 1788666038.582638 write 0x99b000 thru 0x9a6b90 (0xbb91 bytes) 000220[1] 1788666038.604433 write 0x733000 thru 0x740c2d (0xdc2e bytes) 000222[0] 1788666038.801940 mapread 0x051000 thru 0x0594a5 (0x84a6 bytes) 000225[1] 1788666038.962666 mapread 0x86c000 thru 0x8705a8 (0x45a9 bytes) 000226[1] 1788666039.081072 punch from 0x090fe0 to 0x0946f3 (0x3714 bytes) 000227[1] 1788666039.130021 read 0x202000 thru 0x20288f (0x890 bytes) 000229[1] 1788666039.177520 punch from 0x5da310 to 0x5de88a (0x457b bytes) 000230[0] 1788666039.182240 punch from 0x15ee32 to 0x16d2d4 (0xe4a3 bytes) 000231[0] 1788666039.200635 punch from 0x106fc0 to 0x107b54 (0xb95 bytes) 000232[0] 1788666039.205798 mapwrite 0x76a000 thru 0x76c48f (0x2490 bytes) 000235[0] 1788666039.420447 read 0x5ca000 thru 0x5cfb39 (0x5b3a bytes) 000236[0] 1788666039.446575 read 0x063000 thru 0x06c31e (0x931f bytes) 000237[1] 1788666039.466241 mapread 0x2fe000 thru 0x309f44 (0xbf45 bytes) 000238[1] 1788666039.539592 punch from 0x21c930 to 0x22bd4e (0xf41f bytes) 000241[1] 1788666039.607845 punch from 0x1912b9 to 0x19614c (0x4e94 bytes) 000242[0] 1788666039.634666 punch from 0x96d791 to 0x97bc58 (0xe4c8 bytes) 000246[0] 1788666039.678952 mapwrite 0x34a000 thru 0x358329 (0xe32a bytes) 000248[0] 1788666039.785819 punch from 0x8141ae to 0x81788e (0x36e1 bytes) 000249[1] 1788666039.792752 mapread 0x45b000 thru 0x460013 (0x5014 bytes) 000250[1] 1788666039.850202 punch from 0x2e30d0 to 0x2f1b05 (0xea36 bytes) 000251[1] 1788666039.875326 mapread 0x189000 thru 0x18e825 (0x5826 bytes) 000252[1] 1788666039.898071 mapread 0x5c8000 thru 0x5d4b8f (0xcb90 bytes) 000253[0] 1788666039.952397 trunc from 0x9a6b91 to 0x1b23de (0x7f47b2 bytes) 000254[1] 1788666039.999796 punch from 0x175ff6 to 0x177e9c (0x1ea7 bytes) 000255[0] 1788666040.042447 write 0x14b000 thru 0x14e88b (0x388c bytes) 000256[1] 1788666040.064298 read 0x049000 thru 0x051bda (0x8bdb bytes) 000260[1] 1788666040.144832 trunc from 0x1b23df to 0x2e83cc (0x135fee bytes) 000261[1] 1788666040.187437 trunc from 0x2e83cd to 0x7d2b62 (0x4ea796 bytes) 000263[1] 1788666040.208630 punch from 0x51e04d to 0x51f98d (0x1941 bytes) 000264[1] 1788666040.213640 punch from 0x571e26 to 0x577b56 (0x5d31 bytes) 000266[1] 1788666040.256728 read 0x005000 thru 0x0077f4 (0x27f5 bytes) 000267[1] 1788666040.310412 punch from 0x3e0617 to 0x3ecf0d (0xc8f7 bytes) 000269[1] 1788666040.361623 trunc from 0x7d2b63 to 0x6c6485 (0x10c6dd bytes) 000270[0] 1788666040.420316 read 0x42d000 thru 0x436e0a (0x9e0b bytes) 000271[0] 1788666040.459432 read 0x5cf000 thru 0x5d2706 (0x3707 bytes) 000277[1] 1788666040.473168 punch from 0x281831 to 0x285bf3 (0x43c3 bytes) 000279[1] 1788666040.542086 write 0x672000 thru 0x67c895 (0xa896 bytes) 000281[0] 1788666040.574273 read 0x10f000 thru 0x1118dc (0x28dd bytes) 000282[0] 1788666040.613223 read 0x3d9000 thru 0x3d9f49 (0xf4a bytes) 000283[0] 1788666040.620823 trunc from 0x6c6486 to 0x24eeaf (0x4775d6 bytes) 000286[1] 1788666040.707182 trunc from 0x24eeb0 to 0x82a1b8 (0x5db309 bytes) 000287[1] 1788666040.747236 mapwrite 0x1ce000 thru 0x1d1551 (0x3552 bytes) 000288[1] 1788666040.840350 mapwrite 0x9fe000 thru 0x9ffb88 (0x1b89 bytes) 000289[1] 1788666040.913613 write 0x980000 thru 0x98b5a3 (0xb5a4 bytes) 000290[0] 1788666040.926992 write 0x465000 thru 0x470151 (0xb152 bytes) 000292[0] 1788666040.986585 read 0x952000 thru 0x95ffe5 (0xdfe6 bytes) 000293[0] 1788666041.019856 trunc from 0x9ffb89 to 0x85c091 (0x1a3af7 bytes) 000296[0] 1788666041.076039 trunc from 0x85c092 to 0x2c65b6 (0x595adb bytes) 000297[1] 1788666041.121762 mapread 0x1b5000 thru 0x1bea8d (0x9a8e bytes) 000298[1] 1788666041.169653 trunc from 0x2c65b7 to 0x7daac6 (0x514510 bytes) 000299[1] 1788666041.204696 trunc from 0x7daac7 to 0x0865ff (0x7544c7 bytes) 000300[1] 1788666041.220159 read 0x083000 thru 0x0859ba (0x29bb bytes) 000301[0] 1788666041.261050 trunc from 0x086600 to 0x19ee8f (0x118890 bytes) 000302[1] 1788666041.326643 mapwrite 0x2d5000 thru 0x2dc28c (0x728d bytes) 000303[1] 1788666041.505805 mapread 0x0f4000 thru 0x0ffc3d (0xbc3e bytes) 000305[0] 1788666041.530754 mapwrite 0x6b6000 thru 0x6c0093 (0xa094 bytes) 000306[1] 1788666041.737723 punch from 0x229267 to 0x234afe (0xb898 bytes) truncating to largest ever: 0x94eb23 000307[1] 1788666041.762266 trunc from 0x6c0094 to 0x94eb22 (0x28ea8f bytes) 000309[0] 1788666041.786570 read 0x18f000 thru 0x199f12 (0xaf13 bytes) 000311[1] 1788666041.840596 trunc from 0x94eb23 to 0x4a2eda (0x4abc48 bytes) 000312[1] 1788666041.903996 mapread 0x2f9000 thru 0x304bd9 (0xbbda bytes) 000313[1] 1788666041.957146 trunc from 0x4a2edb to 0x55d1af (0xba2d5 bytes) 000314[1] 1788666041.979772 punch from 0x4da506 to 0x4e2a90 (0x858b bytes) 000315[0] 1788666041.985382 mapread 0x0e0000 thru 0x0eacc7 (0xacc8 bytes) 000316[0] 1788666042.060518 read 0x1fa000 thru 0x2033b4 (0x93b5 bytes) 000317[0] 1788666042.084796 trunc from 0x55d1b0 to 0x24e2f5 (0x30eeba bytes) 000318[1] 1788666042.169344 trunc from 0x24e2f6 to 0x2b1671 (0x6337c bytes) 000319[0] 1788666042.215436 write 0x284000 thru 0x286914 (0x2915 bytes) 000320[1] 1788666042.318149 punch from 0x19e3a9 to 0x1a79db (0x9633 bytes) 000321[0] 1788666042.370748 mapwrite 0x029000 thru 0x02bacb (0x2acc bytes) 000322[1] 1788666042.499782 punch from 0x22e736 to 0x23060f (0x1eda bytes) 000323[1] 1788666042.525404 trunc from 0x2b1672 to 0x21cd2a (0x94947 bytes) 000324[1] 1788666042.558292 mapwrite 0x75f000 thru 0x75f5e2 (0x5e3 bytes) 000328[1] 1788666042.713137 trunc from 0x75f5e3 to 0x83d0fa (0xddb18 bytes) 000329[0] 1788666042.755388 punch from 0x313610 to 0x315bc0 (0x25b1 bytes) 000330[0] 1788666042.777238 mapread 0x432000 thru 0x435da8 (0x3da9 bytes) 000331[0] 1788666042.829521 mapwrite 0x7d5000 thru 0x7dcc89 (0x7c8a bytes) 000332[0] 1788666042.934108 punch from 0x6ec02c to 0x6ec933 (0x908 bytes) 000334[1] 1788666042.974046 trunc from 0x83d0fb to 0x138153 (0x704fa7 bytes) 000336[1] 1788666043.013412 write 0x8a7000 thru 0x8a7ae2 (0xae3 bytes) 000337[1] 1788666043.040734 mapread 0x74a000 thru 0x759b87 (0xfb88 bytes) 000338[1] 1788666043.115765 read 0x5f6000 thru 0x5f7d94 (0x1d95 bytes) 000341[1] 1788666043.121615 write 0x1f2000 thru 0x1f95c3 (0x75c4 bytes) 000342[0] 1788666043.141198 punch from 0x61e9fd to 0x62c653 (0xdc57 bytes) 000343[0] 1788666043.171774 mapwrite 0x08a000 thru 0x0980e7 (0xe0e8 bytes) 000345[0] 1788666043.316362 write 0x366000 thru 0x368c91 (0x2c92 bytes) 000346[1] 1788666043.340244 mapread 0x320000 thru 0x323e13 (0x3e14 bytes) 000347[0] 1788666043.400166 read 0x0dd000 thru 0x0e2b74 (0x5b75 bytes) 000348[1] 1788666043.409430 trunc from 0x8a7ae3 to 0x061153 (0x84698f bytes) 000349[0] 1788666043.478099 punch from 0x05dfbc to 0x061153 (0x3198 bytes) 000350[0] 1788666043.524125 mapwrite 0x420000 thru 0x4230d4 (0x30d5 bytes) 000352[0] 1788666043.692996 read 0x262000 thru 0x26f3a6 (0xd3a7 bytes) 000353[0] 1788666043.731543 punch from 0x261be9 to 0x26c44f (0xa867 bytes) 000354[0] 1788666043.769440 write 0x04b000 thru 0x04c703 (0x1704 bytes) 000355[0] 1788666043.792997 mapwrite 0x712000 thru 0x716df6 (0x4df7 bytes) 000356[1] 1788666044.062675 mapread 0x15b000 thru 0x15f504 (0x4505 bytes) 000357[0] 1788666044.131153 read 0x424000 thru 0x430009 (0xc00a bytes) 000361[0] 1788666044.146079 write 0x9bc000 thru 0x9bd2b0 (0x12b1 bytes) 000362[0] 1788666044.179984 trunc from 0x9bd2b1 to 0x0e834a (0x8d4f66 bytes) 000365[1] 1788666044.246118 punch from 0x09a5ac to 0x09d1da (0x2c2f bytes) 000366[1] 1788666044.261715 punch from 0x0af914 to 0x0be143 (0xe830 bytes) 000367[1] 1788666044.280958 mapread 0x00c000 thru 0x01680f (0xa810 bytes) 000368[1] 1788666044.317819 trunc from 0x0e834b to 0x3a4f52 (0x2bcc08 bytes) 000369[0] 1788666044.349070 read 0x00f000 thru 0x01633b (0x733c bytes) 000371[1] 1788666044.520197 trunc from 0x3a4f53 to 0x33138e (0x73bc4 bytes) 000374[1] 1788666044.611620 trunc from 0x33138f to 0x46bc86 (0x13a8f8 bytes) 000376[1] 1788666044.637143 punch from 0x0a1129 to 0x0a7e90 (0x6d68 bytes) 000377[0] 1788666044.663943 trunc from 0x46bc87 to 0x0494bc (0x4227ca bytes) 000378[0] 1788666044.719381 mapwrite 0x87b000 thru 0x88ac75 (0xfc76 bytes) 000379[1] 1788666044.964011 mapread 0x5f1000 thru 0x5fa7dc (0x97dd bytes) 000380[1] 1788666045.057845 write 0x978000 thru 0x97c2df (0x42e0 bytes) 000381[0] 1788666045.104976 mapwrite 0x355000 thru 0x355ac6 (0xac7 bytes) 000382[0] 1788666045.187623 write 0x8f2000 thru 0x8ff9be (0xd9bf bytes) 000383[0] 1788666045.267676 trunc from 0x97c2e0 to 0x86e9ac (0x10d933 bytes) 000385[0] 1788666045.281988 mapread 0x0fd000 thru 0x104436 (0x7437 bytes) 000387[1] 1788666045.353681 write 0x419000 thru 0x41b26a (0x226b bytes) 000391[0] 1788666045.402671 mapread 0x5f9000 thru 0x6050cb (0xc0cc bytes) 000394[0] 1788666045.522072 mapwrite 0x542000 thru 0x54d6cc (0xb6cd bytes) 000395[0] 1788666045.757506 mapwrite 0x875000 thru 0x87775d (0x275e bytes) 000397[1] 1788666045.905394 write 0x727000 thru 0x72af99 (0x3f9a bytes) 000398[1] 1788666045.932007 mapwrite 0x123000 thru 0x131882 (0xe883 bytes) 000400[1] 1788666046.012913 read 0x03f000 thru 0x047e61 (0x8e62 bytes) 000402[1] 1788666046.050677 mapwrite 0x338000 thru 0x3393d1 (0x13d2 bytes) 000403[1] 1788666046.181612 mapwrite 0x98f000 thru 0x99ef7e (0xff7f bytes) 000404[1] 1788666046.355417 mapwrite 0x042000 thru 0x04760b (0x560c bytes) 000406[1] 1788666046.457958 mapwrite 0x83d000 thru 0x848831 (0xb832 bytes) 000410[1] 1788666046.615553 mapwrite 0x0ed000 thru 0x0f0683 (0x3684 bytes) 000411[0] 1788666046.733404 mapread 0x0bf000 thru 0x0c62a5 (0x72a6 bytes) 000412[1] 1788666046.816073 mapwrite 0x7dd000 thru 0x7e6682 (0x9683 bytes) 000413[1] 1788666046.932245 read 0x1a5000 thru 0x1af40c (0xa40d bytes) 000417[0] 1788666046.968239 read 0x159000 thru 0x167b5f (0xeb60 bytes) 000418[1] 1788666047.007715 trunc from 0x99ef7f to 0x7fe508 (0x1a0a76 bytes) 000420[1] 1788666047.086708 trunc from 0x7fe509 to 0x745caf (0xb8859 bytes) 000421[1] 1788666047.149116 read 0x395000 thru 0x3a34ea (0xe4eb bytes) 000422[1] 1788666047.177428 mapwrite 0x472000 thru 0x4752d8 (0x32d9 bytes) 000424[1] 1788666047.348845 read 0x05c000 thru 0x069ebb (0xdebc bytes) 000425[1] 1788666047.373670 punch from 0x0de071 to 0x0e953f (0xb4cf bytes) 000427[1] 1788666047.425914 mapwrite 0x612000 thru 0x61eb3e (0xcb3f bytes) 000428[0] 1788666047.641555 trunc from 0x745cb0 to 0x0dfbc1 (0x6660ee bytes) 000429[0] 1788666047.703015 write 0x1c1000 thru 0x1cca53 (0xba54 bytes) 000431[0] 1788666047.710725 mapwrite 0x3c5000 thru 0x3c704a (0x204b bytes) 000434[0] 1788666047.991067 punch from 0x1ccb8e to 0x1d097a (0x3ded bytes) 000438[0] 1788666048.037339 punch from 0x39fb96 to 0x3a954a (0x99b5 bytes) 000440[0] 1788666048.063110 write 0x9d7000 thru 0x9daf23 (0x3f24 bytes) 000445[1] 1788666048.115366 mapwrite 0x638000 thru 0x64370d (0xb70e bytes) 000446[1] 1788666048.312237 write 0x539000 thru 0x53a8cd (0x18ce bytes) 000447[0] 1788666048.373800 mapread 0x62b000 thru 0x62ba61 (0xa62 bytes) 000448[1] 1788666048.425428 write 0x2c1000 thru 0x2c7ab7 (0x6ab8 bytes) 000450[0] 1788666048.461750 write 0x351000 thru 0x35f7b2 (0xe7b3 bytes) 000451[1] 1788666048.537804 trunc from 0x9daf24 to 0x11e391 (0x8bcb92 bytes) 000454[1] 1788666048.583858 mapwrite 0x265000 thru 0x26bbe1 (0x6be2 bytes) 000455[1] 1788666048.695141 read 0x248000 thru 0x24e704 (0x6705 bytes) 000456[1] 1788666048.766798 trunc from 0x26bbe2 to 0x91ebfd (0x6b301c bytes) 000457[0] 1788666048.792137 punch from 0x172c13 to 0x17930b (0x66f9 bytes) 000458[1] 1788666048.828325 mapread 0x457000 thru 0x457380 (0x381 bytes) 000459[1] 1788666048.937864 trunc from 0x91ebfe to 0x07ed56 (0x89fea7 bytes) 000460[0] 1788666049.024724 read 0x06a000 thru 0x072d99 (0x8d9a bytes) 000461[1] 1788666049.065173 write 0x2b5000 thru 0x2ba8cd (0x58ce bytes) truncating to largest ever: 0x9fa12a 000462[1] 1788666049.100944 trunc from 0x2ba8ce to 0x9fa129 (0x73f85c bytes) 000465[1] 1788666049.175921 trunc from 0x9fa12a to 0x62bf00 (0x3ce229 bytes) 000466[0] 1788666049.212064 read 0x1cd000 thru 0x1da4d9 (0xd4da bytes) 000467[1] 1788666049.270947 write 0x75f000 thru 0x76441e (0x541f bytes) 000470[1] 1788666049.324063 trunc from 0x76441f to 0x40819c (0x35c282 bytes) 000472[0] 1788666049.384040 trunc from 0x40819d to 0x1d3a21 (0x23477b bytes) 000474[1] 1788666049.450922 mapread 0x1a8000 thru 0x1b0aa7 (0x8aa8 bytes) 000475[1] 1788666049.527920 write 0x87a000 thru 0x889f03 (0xff04 bytes) 000478[0] 1788666049.615022 write 0x5c5000 thru 0x5d4765 (0xf766 bytes) 000480[1] 1788666049.735422 mapwrite 0x7ba000 thru 0x7bbeac (0x1ead bytes) 000481[0] 1788666049.962513 punch from 0x44e7a3 to 0x4519a3 (0x3201 bytes) 000484[1] 1788666050.000074 punch from 0x2c48c5 to 0x2cfc76 (0xb3b2 bytes) 000485[1] 1788666050.068093 read 0x5a8000 thru 0x5b60ec (0xe0ed bytes) 000487[1] 1788666050.134212 read 0x526000 thru 0x52dd56 (0x7d57 bytes) 000488[1] 1788666050.149875 write 0x983000 thru 0x98a436 (0x7437 bytes) 000489[1] 1788666050.170370 read 0x246000 thru 0x252114 (0xc115 bytes) 000490[1] 1788666050.188160 mapwrite 0x6eb000 thru 0x6f5f70 (0xaf71 bytes) 000491[0] 1788666050.288882 mapwrite 0x707000 thru 0x70beb9 (0x4eba bytes) 000494[1] 1788666050.450433 read 0x305000 thru 0x30aa96 (0x5a97 bytes) 000495[0] 1788666050.484590 mapread 0x368000 thru 0x3734ba (0xb4bb bytes) 000496[1] 1788666050.519026 trunc from 0x98a437 to 0x5b9c10 (0x3d0826 bytes) 000498[1] 1788666050.576903 write 0x48e000 thru 0x49a5b0 (0xc5b1 bytes) 000500[0] 1788666050.618800 mapwrite 0x65b000 thru 0x667b57 (0xcb58 bytes) All operations completed A-OK! PASS 16a (63s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 23:40:56 (1788666056) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.539699 s, 15.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.290864 s, 28.8 MB/s Chance of close/open is 1 in 50 Seed set to 6061 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1788666062.542818 mapwrite 0x68a80f thru 0x68ba10 (0x1202 bytes) 000006[1] 1788666062.753996 write 0x8c3f29 thru 0x8c4124 (0x1fc bytes) 000007[0] 1788666062.791863 mapread 0x8a3efa thru 0x8a5bca (0x1cd1 bytes) 000010[0] 1788666062.844023 write 0x610aed thru 0x610ebd (0x3d1 bytes) 000011[1] 1788666062.872976 mapread 0x31127f thru 0x312975 (0x16f7 bytes) 000012[1] 1788666062.897218 mapwrite 0x90baa3 thru 0x90ce61 (0x13bf bytes) 000013[0] 1788666063.056238 mapread 0x6e1d14 thru 0x6e2ddb (0x10c8 bytes) 000017[0] 1788666063.161288 mapwrite 0x526e30 thru 0x528675 (0x1846 bytes) 000018[0] 1788666063.300366 write 0x677e0c thru 0x67997b (0x1b70 bytes) 000021[0] 1788666063.405740 punch from 0x01b2cc to 0x01b9c6 (0x6fb bytes) 000022[1] 1788666063.457248 mapwrite 0x78c84e thru 0x78d25c (0xa0f bytes) truncating to largest ever: 0x682f95 000023[0] 1788666063.738353 trunc from 0x90ce62 to 0x682f94 (0x289ecd bytes) 000024[1] 1788666063.860877 write 0x12e870 thru 0x12ee40 (0x5d1 bytes) 000025[1] 1788666063.899864 punch from 0x0aee0e to 0x0af5b0 (0x7a3 bytes) 000026[1] 1788666063.924825 mapwrite 0x436638 thru 0x437a68 (0x1431 bytes) 000027[0] 1788666064.020016 punch from 0x503eda to 0x505c15 (0x1d3c bytes) 000030[1] 1788666064.126411 mapread 0x07bd4f thru 0x07d099 (0x134b bytes) 000034[1] 1788666064.258124 trunc from 0x682f95 to 0x39ca1a (0x2e657a bytes) 000035[0] 1788666064.369062 trunc from 0x39ca1b to 0x0e6af8 (0x2b5f22 bytes) 000036[1] 1788666064.452425 punch from 0x050d6d to 0x0517e1 (0xa75 bytes) 000038[0] 1788666064.501094 write 0x5dd63a thru 0x5ddca7 (0x66e bytes) 000039[0] 1788666064.517104 read 0x2a12a0 thru 0x2a1ef8 (0xc59 bytes) 000040[1] 1788666064.594792 mapread 0x4ea148 thru 0x4ebf28 (0x1de1 bytes) 000041[1] 1788666064.643869 mapwrite 0x75a11e thru 0x75bb87 (0x1a6a bytes) 000042[1] 1788666064.814585 punch from 0x08c647 to 0x08d6e4 (0x109e bytes) 000043[1] 1788666064.853243 read 0x667e45 thru 0x669232 (0x13ee bytes) 000044[1] 1788666064.864855 punch from 0x0f60f0 to 0x0f631b (0x22c bytes) 000045[1] 1788666064.869637 mapwrite 0x85f8b9 thru 0x85fb9b (0x2e3 bytes) 000050[0] 1788666064.986012 trunc from 0x85fb9c to 0x136496 (0x729705 bytes) 000053[0] 1788666065.104858 write 0x648c33 thru 0x6492d4 (0x6a2 bytes) 000054[1] 1788666065.109568 punch from 0x53dcd3 to 0x53f221 (0x154f bytes) 000057[1] 1788666065.259369 write 0x852346 thru 0x85344d (0x1108 bytes) 000062[1] 1788666065.411892 mapwrite 0x8a0f1c thru 0x8a2e00 (0x1ee5 bytes) 000064[0] 1788666065.525842 mapwrite 0x8c3321 thru 0x8c461a (0x12fa bytes) 000065[0] 1788666065.833141 write 0x421fac thru 0x422b65 (0xbba bytes) 000068[1] 1788666066.004318 mapwrite 0x8e70d7 thru 0x8e8a47 (0x1971 bytes) 000069[0] 1788666066.189031 punch from 0x59dbe5 to 0x59ef37 (0x1353 bytes) 000073[0] 1788666066.376874 mapread 0x1caa52 thru 0x1cc261 (0x1810 bytes) 000075[0] 1788666066.442700 read 0x566ebe thru 0x568489 (0x15cc bytes) 000077[1] 1788666066.469186 write 0x975463 thru 0x9771fc (0x1d9a bytes) 000078[0] 1788666066.494692 trunc from 0x9771fd to 0x008910 (0x96e8ec bytes) 000079[1] 1788666066.615378 mapread 0x006eb6 thru 0x008910 (0x1a5b bytes) 000081[0] 1788666066.751315 trunc from 0x008911 to 0x431b46 (0x429236 bytes) 000083[1] 1788666066.873738 punch from 0x2c47bb to 0x2c50e7 (0x92d bytes) 000086[1] 1788666067.026775 read 0x1a4782 thru 0x1a6634 (0x1eb3 bytes) 000088[1] 1788666067.110101 trunc from 0x431b47 to 0x538e28 (0x1072e2 bytes) 000089[1] 1788666067.175827 trunc from 0x538e29 to 0x56ef60 (0x36138 bytes) 000090[1] 1788666067.213874 punch from 0x4e000c to 0x4e0c4d (0xc42 bytes) 000091[0] 1788666067.220164 read 0x490a63 thru 0x4910cd (0x66b bytes) 000092[1] 1788666067.269767 write 0x2268dd thru 0x226a5c (0x180 bytes) 000093[0] 1788666067.323600 punch from 0x17265d to 0x1731cd (0xb71 bytes) 000094[0] 1788666067.380232 mapwrite 0x486b97 thru 0x486d27 (0x191 bytes) 000096[1] 1788666067.517168 write 0x234793 thru 0x235f2b (0x1799 bytes) 000097[1] 1788666067.571946 mapwrite 0x93bd34 thru 0x93ca4a (0xd17 bytes) 000098[1] 1788666067.786021 mapread 0x02b782 thru 0x02c589 (0xe08 bytes) 000099[0] 1788666067.867553 mapwrite 0x9e302e thru 0x9e49a3 (0x1976 bytes) 000100[1] 1788666068.033404 trunc from 0x9e49a4 to 0x33af0d (0x6a9a96 bytes) 000101[1] 1788666068.085040 trunc from 0x33af0e to 0x138134 (0x202dd9 bytes) 000103[1] 1788666068.132816 mapread 0x057ba3 thru 0x0587e5 (0xc43 bytes) 000105[1] 1788666068.186790 write 0x594bdd thru 0x596046 (0x146a bytes) 000106[0] 1788666068.195146 mapwrite 0x50c4e1 thru 0x50cf92 (0xab2 bytes) 000108[0] 1788666068.309225 write 0x51ea20 thru 0x52058e (0x1b6f bytes) 000109[0] 1788666068.341260 mapread 0x13e58e thru 0x13f3b3 (0xe26 bytes) truncating to largest ever: 0x9a2448 000110[1] 1788666068.393863 trunc from 0x596047 to 0x9a2447 (0x40c401 bytes) 000111[1] 1788666068.473636 mapread 0x8cb0cc thru 0x8cbaa1 (0x9d6 bytes) 000112[1] 1788666068.479086 write 0x8f22d9 thru 0x8f236d (0x95 bytes) 000113[1] 1788666068.513649 read 0x328336 thru 0x3299b4 (0x167f bytes) 000114[0] 1788666068.522021 mapread 0x758b74 thru 0x759138 (0x5c5 bytes) 000116[0] 1788666068.591143 punch from 0x98989a to 0x98b4d8 (0x1c3f bytes) 000118[0] 1788666068.617928 punch from 0x71963f to 0x71b534 (0x1ef6 bytes) 000119[0] 1788666068.678967 trunc from 0x9a2448 to 0x813f4f (0x18e4f8 bytes) 000120[0] 1788666068.729804 read 0x50ded0 thru 0x50e4a8 (0x5d9 bytes) 000121[0] 1788666068.787179 mapread 0x0b80d6 thru 0x0b8fe9 (0xf14 bytes) 000122[1] 1788666068.798886 write 0x218d2f thru 0x219058 (0x32a bytes) 000123[1] 1788666068.899776 punch from 0x04adff to 0x04b53e (0x740 bytes) 000124[0] 1788666068.980136 read 0x7d329a thru 0x7d363c (0x3a3 bytes) 000125[0] 1788666069.032853 write 0x9226c4 thru 0x9230b7 (0x9f4 bytes) 000126[0] 1788666069.079368 read 0x6afaa2 thru 0x6afc77 (0x1d6 bytes) 000127[0] 1788666069.156859 write 0x2f57d5 thru 0x2f70a5 (0x18d1 bytes) truncating to largest ever: 0x9e1750 000129[1] 1788666069.176797 trunc from 0x9230b8 to 0x9e174f (0xbe698 bytes) 000131[0] 1788666069.281184 punch from 0x50f8e8 to 0x5116a5 (0x1dbe bytes) 000132[1] 1788666069.356067 mapread 0x84c894 thru 0x84dc68 (0x13d5 bytes) 000133[0] 1788666069.429863 write 0x7742e0 thru 0x77468f (0x3b0 bytes) 000136[0] 1788666069.515605 write 0x4db7e5 thru 0x4dbfad (0x7c9 bytes) 000137[1] 1788666069.546900 trunc from 0x9e1750 to 0x8014f2 (0x1e025d bytes) 000138[1] 1788666069.609007 trunc from 0x8014f3 to 0x2eb830 (0x515cc2 bytes) 000140[1] 1788666069.681176 write 0x510084 thru 0x51054f (0x4cc bytes) 000141[0] 1788666069.687638 trunc from 0x510550 to 0x651d1d (0x1417ce bytes) 000142[0] 1788666069.837370 punch from 0x62a0ea to 0x62b55c (0x1473 bytes) 000145[0] 1788666070.001222 trunc from 0x651d1e to 0x6b2db8 (0x6109b bytes) 000146[1] 1788666070.129634 punch from 0x283493 to 0x28491e (0x148c bytes) 000148[0] 1788666070.257826 mapread 0x005a96 thru 0x006ddf (0x134a bytes) 000149[1] 1788666070.355816 punch from 0x298a4a to 0x2999dd (0xf94 bytes) 000151[1] 1788666070.485222 read 0x24e6cc thru 0x24f58e (0xec3 bytes) 000152[0] 1788666070.550217 mapwrite 0x84919f thru 0x849ae4 (0x946 bytes) 000153[1] 1788666070.749858 punch from 0x0ea714 to 0x0eb103 (0x9f0 bytes) 000154[0] 1788666070.813929 mapwrite 0x1984cc thru 0x198fcf (0xb04 bytes) 000159[0] 1788666070.967992 punch from 0x6d9b86 to 0x6da228 (0x6a3 bytes) 000160[0] 1788666070.972376 write 0x420b5a thru 0x421333 (0x7da bytes) 000163[1] 1788666071.101015 read 0x4f3189 thru 0x4f3736 (0x5ae bytes) 000164[1] 1788666071.147777 mapwrite 0x042ed7 thru 0x043aa4 (0xbce bytes) 000165[0] 1788666071.268499 punch from 0x7de7a2 to 0x7e04ec (0x1d4b bytes) 000166[0] 1788666071.313023 mapwrite 0x42afca thru 0x42b214 (0x24b bytes) 000167[1] 1788666071.460962 write 0x17bcbd thru 0x17c74d (0xa91 bytes) 000169[0] 1788666071.523027 punch from 0x7de73d to 0x7df1ae (0xa72 bytes) 000170[0] 1788666071.551671 write 0x2ef8a3 thru 0x2f0eea (0x1648 bytes) 000171[1] 1788666071.598062 write 0x9fe98b thru 0x9fffff (0x1675 bytes) 000175[0] 1788666071.789887 mapwrite 0x97b58a thru 0x97b8c9 (0x340 bytes) 000180[1] 1788666072.126641 mapwrite 0x607a76 thru 0x6087b9 (0xd44 bytes) 000184[1] 1788666072.385039 punch from 0x138f80 to 0x13a3cc (0x144d bytes) 000185[0] 1788666072.423022 trunc from 0xa00000 to 0x5e14e4 (0x41eb1b bytes) 000188[0] 1788666072.573021 mapwrite 0x0942c1 thru 0x09468c (0x3cc bytes) 000189[1] 1788666072.648976 trunc from 0x5e14e5 to 0x66e110 (0x8cc2c bytes) 000190[0] 1788666072.694214 punch from 0x4eff5a to 0x4f0122 (0x1c9 bytes) 000191[0] 1788666072.730099 trunc from 0x66e111 to 0x90f6e8 (0x2a15d8 bytes) 000192[1] 1788666072.768794 write 0x5a6def thru 0x5a7444 (0x656 bytes) 000193[0] 1788666072.812142 trunc from 0x90f6e9 to 0x79de9e (0x17184a bytes) 000194[1] 1788666072.853225 trunc from 0x79de9f to 0x27b4d0 (0x5229ce bytes) 000195[0] 1788666072.961266 mapread 0x275a54 thru 0x275bde (0x18b bytes) 000196[0] 1788666073.031137 mapwrite 0x73452a thru 0x735af0 (0x15c7 bytes) 000198[1] 1788666073.236163 mapwrite 0x297dba thru 0x297f65 (0x1ac bytes) 000200[1] 1788666073.353196 trunc from 0x735af1 to 0x05852c (0x6dd5c4 bytes) 000201[1] 1788666073.390023 punch from 0x03bf52 to 0x03d1c3 (0x1272 bytes) 000202[1] 1788666073.425360 mapwrite 0x66342f thru 0x663fda (0xbac bytes) 000203[1] 1788666073.541908 write 0x851494 thru 0x852209 (0xd76 bytes) 000204[1] 1788666073.657334 read 0x56fe66 thru 0x5718de (0x1a79 bytes) 000206[1] 1788666073.780970 trunc from 0x85220a to 0x9b7bb3 (0x1659aa bytes) 000209[0] 1788666073.932940 punch from 0x556b72 to 0x5574e0 (0x96f bytes) 000211[0] 1788666074.011073 read 0x7ae841 thru 0x7af0d6 (0x896 bytes) 000212[0] 1788666074.030576 trunc from 0x9b7bb4 to 0x89ef93 (0x118c20 bytes) 000213[0] 1788666074.103034 write 0x205c14 thru 0x206534 (0x921 bytes) 000214[1] 1788666074.123838 punch from 0x58d0cc to 0x58e581 (0x14b6 bytes) 000215[1] 1788666074.167605 write 0x5da2d4 thru 0x5db45d (0x118a bytes) 000216[1] 1788666074.216181 mapread 0x430ba0 thru 0x432236 (0x1697 bytes) 000217[1] 1788666074.278694 trunc from 0x89ef94 to 0x1846ae (0x71a8e5 bytes) 000218[0] 1788666074.367788 mapread 0x075f32 thru 0x076d8d (0xe5c bytes) 000219[1] 1788666074.452926 write 0x95f1e5 thru 0x960e7b (0x1c97 bytes) 000221[1] 1788666074.628649 mapwrite 0x04c913 thru 0x04daa9 (0x1197 bytes) 000223[0] 1788666074.764588 mapread 0x9022f4 thru 0x904258 (0x1f65 bytes) 000225[1] 1788666074.831308 punch from 0x55d8d4 to 0x55e6fb (0xe28 bytes) 000227[0] 1788666074.872015 mapread 0x8f7a84 thru 0x8f7e6f (0x3ec bytes) 000231[1] 1788666074.906590 trunc from 0x960e7c to 0x4c3252 (0x49dc29 bytes) 000235[0] 1788666075.028582 mapread 0x0d78ee thru 0x0d9247 (0x195a bytes) 000239[0] 1788666075.092215 punch from 0x4a51fd to 0x4a5a16 (0x81a bytes) 000241[0] 1788666075.130979 read 0x44f1a6 thru 0x44fe1c (0xc77 bytes) 000242[1] 1788666075.163101 mapwrite 0x12875b thru 0x128c3b (0x4e1 bytes) 000247[0] 1788666075.393837 mapread 0x38ab41 thru 0x38abdd (0x9d bytes) 000248[1] 1788666075.454006 trunc from 0x4c3253 to 0x0c95e0 (0x3f9c72 bytes) 000252[0] 1788666075.661540 mapwrite 0x88989a thru 0x889ed4 (0x63b bytes) 000254[0] 1788666075.849803 mapread 0x6cc967 thru 0x6cd78d (0xe27 bytes) 000255[0] 1788666075.870568 mapread 0x841e93 thru 0x843a75 (0x1be3 bytes) 000256[0] 1788666075.979451 write 0x614504 thru 0x615ff2 (0x1aef bytes) 000257[0] 1788666076.006864 read 0x2380d3 thru 0x238d7c (0xcaa bytes) 000258[0] 1788666076.026126 mapread 0x70d704 thru 0x70e137 (0xa34 bytes) 000260[1] 1788666076.101127 read 0x472a40 thru 0x4730f9 (0x6ba bytes) 000261[0] 1788666076.132017 mapread 0x0fd3ee thru 0x0fdecd (0xae0 bytes) 000262[0] 1788666076.153872 mapread 0x3c0fbc thru 0x3c1f08 (0xf4d bytes) 000263[1] 1788666076.165952 trunc from 0x889ed5 to 0x019c93 (0x870241 bytes) 000264[0] 1788666076.216578 read 0x00c73a thru 0x00e287 (0x1b4e bytes) 000266[0] 1788666076.237889 write 0x99172e thru 0x992779 (0x104c bytes) 000267[0] 1788666076.277612 trunc from 0x99277a to 0x4b52a3 (0x4dd4d6 bytes) 000268[0] 1788666076.386892 mapwrite 0x3883ce thru 0x389e7b (0x1aae bytes) 000270[1] 1788666076.517180 write 0x0efd1d thru 0x0f087f (0xb63 bytes) 000271[0] 1788666076.623450 mapwrite 0x4bc822 thru 0x4be252 (0x1a31 bytes) 000272[0] 1788666076.950719 punch from 0x2dda6a to 0x2de068 (0x5ff bytes) 000273[0] 1788666076.978727 punch from 0x01703f to 0x0187ea (0x17ac bytes) 000274[0] 1788666077.049698 punch from 0x02c7ee to 0x02d6a8 (0xebb bytes) 000275[0] 1788666077.059454 write 0x367cb7 thru 0x367f2e (0x278 bytes) 000276[0] 1788666077.075606 mapwrite 0x0556bb thru 0x055b94 (0x4da bytes) 000278[0] 1788666077.262085 mapread 0x26d644 thru 0x26e3f6 (0xdb3 bytes) 000279[1] 1788666077.282892 mapread 0x479c7d thru 0x47aa5f (0xde3 bytes) 000280[0] 1788666077.370266 read 0x1474eb thru 0x147790 (0x2a6 bytes) 000282[1] 1788666077.448510 write 0x6eb445 thru 0x6ed314 (0x1ed0 bytes) 000283[0] 1788666077.487096 trunc from 0x6ed315 to 0x53c71a (0x1b0bfa bytes) 000284[1] 1788666077.533557 punch from 0x150903 to 0x15176f (0xe6d bytes) 000286[1] 1788666077.587769 trunc from 0x53c71b to 0x3f4609 (0x148111 bytes) 000287[1] 1788666077.637049 mapread 0x29349a thru 0x293c2b (0x792 bytes) 000291[0] 1788666077.724213 mapwrite 0x75cad5 thru 0x75d6bc (0xbe8 bytes) 000292[1] 1788666077.937871 mapread 0x6bf86a thru 0x6bfe03 (0x59a bytes) 000294[0] 1788666078.008095 punch from 0x36a053 to 0x36aeea (0xe98 bytes) 000295[1] 1788666078.017614 write 0x919c07 thru 0x91a051 (0x44b bytes) 000296[1] 1788666078.091415 write 0x93b014 thru 0x93bdaf (0xd9c bytes) 000297[1] 1788666078.131021 mapwrite 0x22e68d thru 0x22f520 (0xe94 bytes) 000299[1] 1788666078.310956 mapread 0x936d1f thru 0x9382a0 (0x1582 bytes) 000304[0] 1788666078.448971 mapwrite 0x074aad thru 0x074d35 (0x289 bytes) 000305[0] 1788666078.642075 punch from 0x26e7fd to 0x26ee55 (0x659 bytes) 000307[0] 1788666078.663314 mapread 0x0e5558 thru 0x0e57c4 (0x26d bytes) 000308[1] 1788666078.685263 trunc from 0x93bdb0 to 0x13d8ef (0x7fe4c0 bytes) 000311[1] 1788666078.880621 read 0x096523 thru 0x0982a5 (0x1d83 bytes) 000312[0] 1788666079.005867 mapread 0x035bba thru 0x036f0f (0x1356 bytes) 000315[1] 1788666079.238442 mapwrite 0x88d4a5 thru 0x88f320 (0x1e7c bytes) 000316[0] 1788666079.642956 read 0x36d808 thru 0x36f551 (0x1d4a bytes) 000320[0] 1788666079.703244 read 0x3d8572 thru 0x3d92d4 (0xd63 bytes) 000323[0] 1788666079.786322 mapread 0x84a772 thru 0x84b530 (0xdbf bytes) 000325[0] 1788666079.844683 trunc from 0x88f321 to 0x6ca4e8 (0x1c4e38 bytes) 000326[0] 1788666079.977065 write 0x1e4edf thru 0x1e6c24 (0x1d46 bytes) 000329[0] 1788666080.103324 mapwrite 0x0188c2 thru 0x019fe0 (0x171f bytes) 000330[1] 1788666080.303279 punch from 0x4895d1 to 0x4899a5 (0x3d5 bytes) 000334[1] 1788666080.473703 mapwrite 0x1448c6 thru 0x145146 (0x881 bytes) 000336[1] 1788666080.716901 write 0x1a48c0 thru 0x1a67e3 (0x1f24 bytes) 000337[1] 1788666080.741234 punch from 0x3f6fe2 to 0x3f81ad (0x11cc bytes) 000340[1] 1788666080.787103 trunc from 0x6ca4e9 to 0x3971fc (0x3332ec bytes) 000341[1] 1788666080.885915 trunc from 0x3971fd to 0x7aae95 (0x413c99 bytes) 000343[0] 1788666081.061090 mapwrite 0x50dd3d thru 0x50e69b (0x95f bytes) 000346[1] 1788666081.304777 write 0x67613e thru 0x676b02 (0x9c5 bytes) 000347[1] 1788666081.371589 trunc from 0x7aae96 to 0x0f8fea (0x6b1eab bytes) 000348[1] 1788666081.423879 mapread 0x015b66 thru 0x01638f (0x82a bytes) 000349[0] 1788666081.449427 write 0x12e169 thru 0x12f674 (0x150c bytes) 000351[1] 1788666081.515790 mapread 0x0cb9e1 thru 0x0ccf3e (0x155e bytes) 000352[0] 1788666081.615869 mapread 0x001575 thru 0x0030f7 (0x1b83 bytes) 000353[0] 1788666081.689023 write 0x1b20ce thru 0x1b3309 (0x123c bytes) 000354[0] 1788666081.717981 mapread 0x05ff9f thru 0x0614a0 (0x1502 bytes) 000356[0] 1788666081.798762 punch from 0x0c6633 to 0x0c6dfe (0x7cc bytes) 000359[0] 1788666081.882033 trunc from 0x1b330a to 0x69ad82 (0x4e7a79 bytes) 000361[1] 1788666081.967739 trunc from 0x69ad83 to 0x07e363 (0x61ca1f bytes) 000364[0] 1788666082.142354 mapread 0x033a7a thru 0x034b8c (0x1113 bytes) 000365[0] 1788666082.310509 mapwrite 0x69425c thru 0x6946f5 (0x49a bytes) 000366[0] 1788666082.540921 mapread 0x312410 thru 0x312cae (0x89f bytes) 000368[0] 1788666082.592283 read 0x3654d2 thru 0x366d3f (0x186e bytes) 000370[1] 1788666082.650344 mapread 0x3f227a thru 0x3f3a7d (0x1804 bytes) 000371[0] 1788666082.763303 trunc from 0x6946f6 to 0x13b2e3 (0x559412 bytes) 000372[1] 1788666082.802912 trunc from 0x13b2e4 to 0x1143bb (0x26f28 bytes) 000374[0] 1788666082.887432 punch from 0x0331e9 to 0x0345d5 (0x13ed bytes) 000376[1] 1788666082.957909 punch from 0x0647ff to 0x065d7a (0x157c bytes) 000377[1] 1788666083.035190 trunc from 0x1143bc to 0x401ed0 (0x2edb15 bytes) 000380[1] 1788666083.176194 mapwrite 0x746da4 thru 0x7476d3 (0x930 bytes) 000382[1] 1788666083.414148 mapread 0x342a60 thru 0x3436e2 (0xc83 bytes) 000383[0] 1788666083.443930 write 0x6ec643 thru 0x6ecb18 (0x4d6 bytes) 000385[1] 1788666083.560369 trunc from 0x7476d4 to 0x793843 (0x4c170 bytes) 000387[0] 1788666083.697147 trunc from 0x793844 to 0x58a107 (0x20973c bytes) 000388[0] 1788666083.769288 mapwrite 0x98d762 thru 0x98d7e2 (0x81 bytes) 000391[1] 1788666083.962322 read 0x508360 thru 0x508788 (0x429 bytes) 000393[1] 1788666084.025908 mapread 0x6ead1b thru 0x6ec36d (0x1653 bytes) 000397[1] 1788666084.083504 read 0x4b1a73 thru 0x4b2f57 (0x14e5 bytes) 000399[1] 1788666084.110762 punch from 0x27372f to 0x274b9c (0x146e bytes) 000401[0] 1788666084.145413 mapwrite 0x8644be thru 0x865671 (0x11b4 bytes) 000402[0] 1788666084.395792 trunc from 0x98d7e3 to 0x5e68ae (0x3a6f34 bytes) 000406[0] 1788666084.552008 read 0x344695 thru 0x34504e (0x9ba bytes) 000407[0] 1788666084.560764 read 0x114026 thru 0x115d3e (0x1d19 bytes) 000408[0] 1788666084.569411 mapwrite 0x22f2b5 thru 0x230f38 (0x1c84 bytes) 000409[0] 1788666084.703450 read 0x52541d thru 0x52624b (0xe2f bytes) 000410[0] 1788666084.756019 mapwrite 0x1d956e thru 0x1db35f (0x1df2 bytes) 000413[0] 1788666085.023887 mapread 0x29505f thru 0x296ea2 (0x1e44 bytes) 000414[1] 1788666085.048448 trunc from 0x5e68af to 0x3a44ac (0x242402 bytes) 000418[1] 1788666085.196185 mapread 0x2c24bd thru 0x2c28e4 (0x428 bytes) 000419[0] 1788666085.208680 write 0x60a48c thru 0x60c1b1 (0x1d26 bytes) 000420[0] 1788666085.277287 read 0x0d285c thru 0x0d3bea (0x138f bytes) 000421[1] 1788666085.327391 mapread 0x21b962 thru 0x21d779 (0x1e18 bytes) 000422[0] 1788666085.377562 mapread 0x5c2ba5 thru 0x5c4458 (0x18b4 bytes) 000424[1] 1788666085.450386 mapread 0x00ca7a thru 0x00de8b (0x1412 bytes) 000425[0] 1788666085.504302 mapwrite 0x1c9f85 thru 0x1cb5e3 (0x165f bytes) 000426[0] 1788666085.627436 mapread 0x004508 thru 0x005550 (0x1049 bytes) 000427[0] 1788666085.642765 read 0x3e6cf4 thru 0x3e760e (0x91b bytes) 000429[1] 1788666085.651798 read 0x0c7329 thru 0x0c7851 (0x529 bytes) 000432[0] 1788666085.674539 mapwrite 0x0d6806 thru 0x0d6f74 (0x76f bytes) 000433[1] 1788666085.860850 write 0x225692 thru 0x225892 (0x201 bytes) 000434[0] 1788666085.915115 punch from 0x34fdd3 to 0x351bc6 (0x1df4 bytes) 000436[1] 1788666085.988162 punch from 0x2eca6a to 0x2ecc77 (0x20e bytes) 000437[0] 1788666086.031430 read 0x2aa7b5 thru 0x2ab395 (0xbe1 bytes) 000438[1] 1788666086.055574 mapread 0x1725eb thru 0x1740e1 (0x1af7 bytes) 000439[0] 1788666086.084677 trunc from 0x60c1b2 to 0x946bdf (0x33aa2e bytes) 000440[1] 1788666086.167984 mapwrite 0x36dd7c thru 0x36dede (0x163 bytes) 000441[0] 1788666086.287169 mapread 0x1caa33 thru 0x1cc698 (0x1c66 bytes) 000442[1] 1788666086.359123 trunc from 0x946be0 to 0x74b82a (0x1fb3b5 bytes) 000443[1] 1788666086.438725 mapwrite 0x1ab876 thru 0x1ad3fb (0x1b86 bytes) 000445[1] 1788666086.563705 trunc from 0x74b82b to 0x9be796 (0x272f6c bytes) 000446[1] 1788666086.587193 mapread 0x6b1891 thru 0x6b1d4c (0x4bc bytes) 000447[0] 1788666086.604556 mapwrite 0x8ee2b5 thru 0x8eeedd (0xc29 bytes) 000448[1] 1788666086.797323 mapwrite 0x6b916b thru 0x6b91a2 (0x38 bytes) 000449[0] 1788666086.890433 mapread 0x325f48 thru 0x326700 (0x7b9 bytes) 000450[1] 1788666086.923254 trunc from 0x9be797 to 0x30b4ba (0x6b32dc bytes) 000451[0] 1788666086.953276 read 0x21d1d0 thru 0x21e0c7 (0xef8 bytes) 000455[1] 1788666087.027250 mapread 0x172a74 thru 0x173737 (0xcc4 bytes) 000456[1] 1788666087.050872 punch from 0x2ba887 to 0x2bbf05 (0x167f bytes) 000457[1] 1788666087.067914 trunc from 0x30b4bb to 0x881c8e (0x5767d4 bytes) 000460[0] 1788666087.117167 trunc from 0x881c8f to 0x6f9151 (0x188b3d bytes) 000463[0] 1788666087.197077 mapwrite 0x1f5f1b thru 0x1f7714 (0x17fa bytes) 000465[1] 1788666087.332872 mapwrite 0x92109c thru 0x9219c0 (0x925 bytes) 000467[0] 1788666087.547417 punch from 0x75d535 to 0x75f17a (0x1c46 bytes) 000468[0] 1788666087.601167 read 0x0517a0 thru 0x05314c (0x19ad bytes) 000471[1] 1788666087.670387 punch from 0x591f51 to 0x591f96 (0x46 bytes) 000472[1] 1788666087.732710 punch from 0x6462b1 to 0x647a55 (0x17a5 bytes) 000473[1] 1788666087.786043 mapread 0x305b91 thru 0x305c2d (0x9d bytes) 000474[1] 1788666087.836405 punch from 0x65b861 to 0x65c664 (0xe04 bytes) 000475[0] 1788666087.841639 mapread 0x785ec0 thru 0x7860f7 (0x238 bytes) 000476[1] 1788666087.888011 trunc from 0x9219c1 to 0x176c2a (0x7aad96 bytes) 000477[1] 1788666087.938007 read 0x060d46 thru 0x062024 (0x12df bytes) 000478[1] 1788666088.003555 mapwrite 0x680348 thru 0x680a82 (0x73b bytes) 000479[1] 1788666088.184876 write 0x2d4876 thru 0x2d5fb9 (0x1744 bytes) 000480[0] 1788666088.190701 trunc from 0x680a83 to 0x3a6c9f (0x2d9de3 bytes) 000481[0] 1788666088.278221 mapread 0x2dd20a thru 0x2dd2dd (0xd4 bytes) 000482[1] 1788666088.288633 read 0x36b18a thru 0x36b9fa (0x871 bytes) 000484[1] 1788666088.395953 punch from 0x064be9 to 0x066355 (0x176d bytes) 000488[1] 1788666088.488302 read 0x2ba791 thru 0x2bc6cd (0x1f3d bytes) 000489[1] 1788666088.601433 mapread 0x0434ad thru 0x04483f (0x1393 bytes) 000490[1] 1788666088.647597 mapwrite 0x7e63a4 thru 0x7e75de (0x123b bytes) 000491[0] 1788666088.861843 read 0x672db1 thru 0x673b5c (0xdac bytes) 000492[1] 1788666088.958028 read 0x6c1acb thru 0x6c3a38 (0x1f6e bytes) 000493[1] 1788666088.996490 write 0x4aadf5 thru 0x4aaf76 (0x182 bytes) 000495[0] 1788666089.039175 trunc from 0x7e75df to 0x2a8ff9 (0x53e5e5 bytes) 000496[1] 1788666089.202708 mapread 0x021e9a thru 0x0229db (0xb42 bytes) 000497[0] 1788666089.300292 mapread 0x1b1cab thru 0x1b379d (0x1af3 bytes) 000498[0] 1788666089.367582 mapwrite 0x056aae thru 0x057e07 (0x135a bytes) 000499[0] 1788666089.480654 mapread 0x0b0143 thru 0x0b1950 (0x180e bytes) 000500[1] 1788666089.531561 write 0x8129ae thru 0x813d65 (0x13b8 bytes) All operations completed A-OK! PASS 16b (39s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 23:41:35 (1788666095) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 23:41:40 (1788666100) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.664055 s, 12.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.766628 s, 10.9 MB/s ldlm.namespaces.MGC192.168.204.122@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9229c84b7800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9229c96ec800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9229c84b7800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9229c84b7800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 17.6356 s, 5.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.53982 s, 23.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.63371 s, 28.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.34167 s, 14.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8424 s, 9.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.45509 s, 30.3 MB/s PASS 16d (65s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 23:42:45 (1788666165) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.409896 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151548 s, 6.9 MB/s PASS 16e (9s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 23:42:54 (1788666174) TIMEOUT PASS 16f (30s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 23:43:24 (1788666204) TIMEOUT PASS 16g (30s) == sanityn test 16h: mmap read after truncate file ======= 23:43:54 (1788666234) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 23:44:02 (1788666242) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 23:44:10 (1788666250) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.48336 s, 17.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57946 s, 16.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50816 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.8495 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43676 s, 10.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.82565 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.18953 s, 8.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56959 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09207 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.5576 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.92731 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.27015 s, 11.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.65929 s, 9.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.54284 s, 7.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.76105 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.04899 s, 8.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.37682 s, 7.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.44624 s, 5.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.5072 s, 5.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (44s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 23:44:54 (1788666294) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Sat Sep 5 23:45:46 EDT 2026 + fsx 2 done: rc=0 Sat Sep 5 23:45:46 EDT 2026 + fsx 3 done: rc=0 Sat Sep 5 23:45:46 EDT 2026 + fsx 4 done: rc=0 Sat Sep 5 23:45:46 EDT 2026 + fsx 5 done: rc=0 Sat Sep 5 23:45:46 EDT 2026 PASS 16k (60s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 23:45:55 (1788666355) fail_loc=0x8000030a PASS 17 (14s) == sanityn test 18: mmap sanity check =========================================================================================== 23:46:09 (1788666369) mmap test1: basic mmap operation (PASS, 0.21213s) mmap test2: MAP_PRIVATE not write back (PASS, 0.16306s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.183s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.8175s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.2471s) mmap test6: check mmap write/read content on two nodes (PASS, 0.23504s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1398s) mmap test11: multi-thread shared mmap (PASS, 1.2046s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1387s) PASS 18 (33s) == sanityn test 19: test concurrent uncached read races ========================================================================= 23:46:41 (1788666401) oleg422-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 23:46:46 (1788666406) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 23:46:53 (1788666413) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 23:47:01 (1788666421) now is 1788666424 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 new atime is 1788666485 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 23:48:12 (1788666492) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (9s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 23:48:21 (1788666501) PASS 24b (9s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 23:48:30 (1788666510) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (9s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 23:48:39 (1788666519) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 23:48:42 (1788666522) utime: good mknod times 1788666523 <= 1788666524 <= 1788666524 for /mnt/lustre/f26a utime: good mknod times 1788666523 <= 1788666524 <= 1788666524 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 23:48:50 (1788666530) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 23:49:00 (1788666540) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 23:49:02.000000000 -0400 Birth: 2026-09-05 23:49:02.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-05 23:49:02.000000000 -0400 Birth: 2026-09-05 23:49:02.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 23:49:09 (1788666549) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 23:49:18 (1788666558) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 23:49:26 (1788666566) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.122@tcp:/lustre 7542784 38912 7499776 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 23:49:50 (1788666590) -rw-r--r-- 1 root root 0 Sep 5 23:49 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 23:50:00 (1788666600) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43163 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 23:50:09 (1788666609) PASS 31t (9s) == sanityn test 32b: lockless i/o ======================== 23:50:17 (1788666617) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 23:50:21 (1788666621) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.53364 s, 34.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47253 s, 35.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72334 s, 30.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93729 s, 27.1 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.03861 s, 25.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.50851 s, 20.9 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32469 s, 39.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.35988 s, 38.6 MB/s PASS 32c (32s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 23:50:54 (1788666654) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 23:50:58 (1788666658) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 23:51:01 (1788666661) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 23:51:05 (1788666665) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 23:51:08 (1788666668) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid in FULL state after 0 sec oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid in IDLE state after 0 sec oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid in FULL state after 0 sec oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid in IDLE state after 0 sec oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9229c84b7800.ost_server_uuid in FULL state after 0 sec oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9229c84b7800.ost_server_uuid in IDLE state after 0 sec PASS 34 (147s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 23:53:35 (1788666815) fail_loc=0x80000317 Race attempt 0 Wait for 50475 50488 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50475 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.122@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.204.122@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 - open/close 579 (time 1788666833.35 total 10.00 last 57.90) - open/close 1156 (time 1788666843.36 total 20.00 last 57.67) - open/close 1758 (time 1788666853.36 total 30.01 last 60.17) - open/close 2379 (time 1788666863.37 total 40.02 last 62.07) - open/close 2980 (time 1788666873.38 total 50.02 last 60.05) - open/close 3564 (time 1788666883.38 total 60.03 last 58.39) - open/close 3991 (time 1788666893.39 total 70.04 last 42.64) total: 4000 open/close in 70.23 seconds: 56.96 ops/second PASS 35 (78s) == sanityn test 36: handle ESTALE/open-unlink correctly == 23:54:53 (1788666893) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg422-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.740988 s, 70.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7428096) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.903278 s, 58.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7499776) after_dd(7430144) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.830624 s, 63.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7437312) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.889899 s, 58.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7498752) after_dd(7431168) after(7482368) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.884235 s, 59.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7482368) after_dd(7435264) after(7486464) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.861214 s, 60.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7486464) after_dd(7440384) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.851424 s, 61.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7498752) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07184 s, 48.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7488512) after_dd(7438336) after(7498752) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.979779 s, 53.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7498752) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.621624 s, 84.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7437312) after(7488512) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.648221 s, 80.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7488512) after_dd(7431168) after(7482368) PASS 36 (359s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 00:00:52 (1788667252) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 - create 1120 (time 1788667267.22 total 10.02 last 111.79) - create 2327 (time 1788667277.22 total 20.03 last 120.62) - create 3523 (time 1788667287.23 total 30.03 last 119.54) - create 4753 (time 1788667297.23 total 40.04 last 122.95) - create 5918 (time 1788667307.23 total 50.04 last 116.49) - create 6948 (time 1788667317.25 total 60.05 last 102.88) - create 7921 (time 1788667327.26 total 70.06 last 97.16) - create 9022 (time 1788667337.26 total 80.06 last 110.09) total: 10000 create in 88.24 seconds: 113.33 ops/second PASS 37 (102s) == sanityn test 39a: file mtime does not change after rename ========================================================== 00:02:34 (1788667354) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 00:02:44 (1788667364) repeat after cancel_lru_locks PASS 39b (12s) == sanityn test 39c: check truncate mtime update ================================================================================ 00:02:56 (1788667376) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 00:03:05 (1788667385) fail_loc=0x411 fail_loc=0 PASS 39d (9s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 00:03:16 (1788667396) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 00:03:34 (1788667414) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 00:03:54 (1788667434) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 00:04:14 (1788667454) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 00:04:32 (1788667472) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (16s) == sanityn test 41b: pdirops: create vs create ======================================================================== 00:04:48 (1788667488) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 00:05:02 (1788667502) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 00:05:17 (1788667517) Conflict PASS 41d (16s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 00:05:34 (1788667534) Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 00:05:50 (1788667550) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 00:06:05 (1788667565) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 00:06:20 (1788667580) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (18s) == sanityn test 41i: reint_open: create vs create ======== 00:06:38 (1788667598) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1136s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 00:25:34 (1788668734) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 00:25:50 (1788668750) open: File exists Conflict PASS 42b (17s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 00:26:07 (1788668767) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 00:26:23 (1788668783) Conflict PASS 42d (16s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 00:26:38 (1788668798) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 00:26:52 (1788668812) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 00:27:08 (1788668828) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 00:27:22 (1788668842) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 00:27:37 (1788668857) PASS 43a (74s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 00:28:50 (1788668930) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 00:29:04 (1788668944) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 00:29:20 (1788668960) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (17s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 00:29:37 (1788668977) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 00:29:51 (1788668991) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 00:30:06 (1788669006) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 00:30:20 (1788669020) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 00:30:35 (1788669035) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 00:30:38 (1788669038) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (159s) == sanityn test 43k: unlink vs create ==================== 00:33:18 (1788669198) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1247s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 00:54:06 (1788670446) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Sun Sep 6 00:54:15 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 00:54:20 (1788670460) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 00:54:34 (1788670474) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 00:54:50 (1788670490) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 00:55:03 (1788670503) Conflict PASS 44e (18s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 00:55:21 (1788670521) Conflict PASS 44f (16s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 00:55:37 (1788670537) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 00:55:52 (1788670552) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 00:56:07 (1788670567) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 00:56:11 (1788670571) PASS 45a (129s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 00:58:20 (1788670700) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 00:58:35 (1788670715) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 00:58:51 (1788670731) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 00:59:06 (1788670746) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 00:59:21 (1788670761) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 00:59:36 (1788670776) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (14s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 00:59:50 (1788670790) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 01:00:03 (1788670803) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 01:00:07 (1788670807) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1315s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 01:22:02 (1788672122) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (14s) == sanityn test 46b: pdirops: link vs create ======================================================================== 01:22:16 (1788672136) open: File exists Conflict PASS 46b (17s) == sanityn test 46c: pdirops: link vs link ======================================================================== 01:22:33 (1788672153) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (14s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 01:22:47 (1788672167) Conflict PASS 46d (15s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 01:23:02 (1788672182) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 01:23:16 (1788672196) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 01:23:29 (1788672209) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 01:23:43 (1788672223) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 01:23:57 (1788672237) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 01:24:00 (1788672240) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 01:24:04 (1788672244) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 01:24:08 (1788672248) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 01:24:11 (1788672251) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 01:24:14 (1788672254) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 01:24:17 (1788672257) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 01:24:21 (1788672261) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 01:24:24 (1788672264) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.174088 s, 58.8 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 01:24:36 (1788672276) 0+1 records in 0+1 records out 158 bytes copied, 0.0485473 s, 3.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (219022) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 01:24:46 (1788672286) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0145713 s, 70.3 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947413 s, 11.1 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 01:25:07 (1788672307) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0208594 s, 49.1 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 01:25:20 (1788672320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035654 s, 29.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 01:25:29 (1788672329) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2455 0x997 0x240000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 01:25:38 (1788672338) llite.lustre-ffff9229c84b7800.xattr_cache=1 llite.lustre-ffff9229c96ec800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:40.000000000 -0400 Modify: 2026-09-06 01:25:40.000000000 -0400 Change: 2026-09-06 01:25:40.000000000 -0400 Birth: 2026-09-06 01:25:40.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:46.000000000 -0400 Modify: 2026-09-06 01:25:47.000000000 -0400 Change: 2026-09-06 01:25:47.000000000 -0400 Birth: 2026-09-06 01:25:46.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:52.000000000 -0400 Modify: 2026-09-06 01:25:53.000000000 -0400 Change: 2026-09-06 01:25:53.000000000 -0400 Birth: 2026-09-06 01:25:52.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:52.000000000 -0400 Modify: 2026-09-06 01:25:53.000000000 -0400 Change: 2026-09-06 01:25:53.000000000 -0400 Birth: 2026-09-06 01:25:52.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:59.000000000 -0400 Modify: 2026-09-06 01:26:00.000000000 -0400 Change: 2026-09-06 01:26:00.000000000 -0400 Birth: 2026-09-06 01:25:59.000000000 -0400 File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 01:25:59.000000000 -0400 Modify: 2026-09-06 01:26:00.000000000 -0400 Change: 2026-09-06 01:26:00.000000000 -0400 Birth: 2026-09-06 01:25:59.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 01:26:12 (1788672372) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 01:26:26 (1788672386) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 01:26:39 (1788672399) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 01:26:59 (1788672419) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 01:27:15 (1788672435) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 01:27:19 (1788672439) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (22s) == sanityn test 55g: rename: race with trylock =========== 01:27:41 (1788672461) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (27s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 01:28:09 (1788672489) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1788672491 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.540169 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (122s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 01:30:11 (1788672611) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1788672617 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.06151 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (131s) == sanityn test 60: Verify data_version behaviour ======== 01:32:22 (1788672742) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 01:32:32 (1788672752) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 01:32:42 (1788672762) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 01:32:51 (1788672771) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0134198 s, 4.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 01:32:56 (1788672776) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.017887 s, 3.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 01:33:00 (1788672780) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 01:33:05 (1788672785) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0141763 s, 4.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 01:33:08 (1788672788) llite.lustre-ffff9229c84b7800.xattr_cache=1 llite.lustre-ffff9229c96ec800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 01:33:17 (1788672797) llite.lustre-ffff9229c84b7800.xattr_cache=1 llite.lustre-ffff9229c96ec800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 01:33:26 (1788672806) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0198159 s, 51.7 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0128383 s, 79.8 kB/s 231851: get lock2 231852: get lock1 231851: done 231851: get lock1 231852: done 231852: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231852: cannot get lock2: Resource deadlock avoided 231852: exit rc=0 231851: done 231851: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 01:33:39 (1788672819) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0437246 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330454 s, 31.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232437 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 01:33:52 (1788672832) opening files: ........ closing files: ........ PASS 76 (68s) == sanityn test 77a: check FIFO NRS policy =============== 01:35:00 (1788672900) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275362 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729225 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588131 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111877 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149885 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155782 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164152 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182994 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147477 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163012 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165286 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14046 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139999 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192149 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198092 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208541 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218574 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901211 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558121 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570705 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719692 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775636 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11089 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897524 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704379 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949105 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866359 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922425 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985572 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992425 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135201 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105901 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178587 s, 5.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 01:35:11 (1788672911) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.304314 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669092 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775225 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182305 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355127 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188809 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127525 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133312 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215988 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242333 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201071 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293537 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183355 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260583 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186284 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193724 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283598 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303095 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459657 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329821 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666485 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361185 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035286 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485908 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118472 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437171 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046235 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574393 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775885 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084135 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774759 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850721 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998528 s, 10.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.389461 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564451 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914507 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523709 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142014 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794173 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621069 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15236 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722029 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332322 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218052 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711742 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219424 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261105 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228222 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166676 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237286 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795355 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421347 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837237 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758659 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803846 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115318 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851297 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537703 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149207 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575159 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979241 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122292 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063354 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06506 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105702 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665037 s, 15.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 01:35:25 (1788672925) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255453 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408311 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587524 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290108 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104892 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115597 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665003 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108538 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938503 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824793 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098661 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171826 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924619 s, 11.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107603 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132073 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132572 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16413 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024185 s, 434 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941836 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671424 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706061 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936621 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904387 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531761 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557205 s, 18.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567761 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949089 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528646 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150266 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452007 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292296 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408616 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370245 s, 28.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315691 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442941 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073747 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103759 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10012 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494798 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916045 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126428 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097437 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227404 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160226 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182113 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194697 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173814 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129128 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137686 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222591 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084762 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038488 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05239 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966598 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443829 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969114 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462428 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100499 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517411 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511148 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603294 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105579 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689353 s, 15.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674517 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742572 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076249 s, 13.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315071 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247332 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448197 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053753 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458053 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041364 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578619 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402696 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940997 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726346 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954563 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13788 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10356 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681999 s, 15.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690363 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751156 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129198 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487455 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572789 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743904 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770018 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504368 s, 20.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649714 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719029 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728759 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608101 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769745 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874156 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686297 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802861 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821875 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681308 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932602 s, 11.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (16s) == sanityn test 77d: check TRR nrs policy ================ 01:35:41 (1788672941) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313529 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046093 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130326 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128044 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160237 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150665 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136341 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222034 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209785 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213769 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288299 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247713 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259087 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247841 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348043 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3907 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406055 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120201 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971536 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182389 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666436 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158205 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768791 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668607 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109125 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166589 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173918 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219305 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539954 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177997 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371572 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511503 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193648 s, 5.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.336416 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599877 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580157 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108059 s, 9.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122356 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129138 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161519 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182301 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1965 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225495 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200926 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231697 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252086 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232356 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252579 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305691 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424249 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039931 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112052 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090369 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153352 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108454 s, 96.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195151 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890628 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854493 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077544 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654015 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793152 s, 13.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625191 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054597 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510761 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426846 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355048 s, 29.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.355212 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847502 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878789 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166812 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195909 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102176 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118127 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213448 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118726 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20874 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192524 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182109 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231263 s, 4.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257457 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173659 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267046 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232979 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719102 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667654 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850214 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10335 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787337 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851463 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991917 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110712 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863529 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814057 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133985 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13769 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148813 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125271 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196669 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155948 s, 6.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (20s) == sanityn test 77e: check TBF NID nrs policy ============ 01:36:01 (1788672961) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.22@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2437 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517547 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299379 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660862 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930928 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106427 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106342 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154624 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14862 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15265 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168183 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192823 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171819 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183291 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219943 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202551 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234581 s, 4.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910467 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106394 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950211 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859197 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101312 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685148 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110738 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079583 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079937 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991509 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900191 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11151 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956219 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859328 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952863 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107612 s, 9.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.382157 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322346 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098378 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920479 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741199 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035964 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118774 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346241 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754442 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171784 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139175 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982597 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156882 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224159 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951275 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235188 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229898 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708803 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146867 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104635 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150951 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712985 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133696 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136857 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134963 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185628 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140857 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729924 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186235 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105415 s, 99.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147891 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149662 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326183 s, 32.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.347081 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527317 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657114 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737204 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072951 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140946 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190219 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192846 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162031 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160812 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139498 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16625 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201392 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244054 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219787 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22899 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231222 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044711 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483019 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12749 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126197 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809547 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709914 s, 14.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902346 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929628 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12489 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790841 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837742 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103733 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107115 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889659 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921979 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978102 s, 10.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.333998 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042801 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577875 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669752 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434776 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728871 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522232 s, 20.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749298 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761125 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249164 s, 42.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177532 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186647 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220801 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229903 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235927 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232263 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256448 s, 4.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088303 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117397 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966155 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00476357 s, 220 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109351 s, 9.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116213 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00774217 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119368 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850866 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118006 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119512 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765202 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123781 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112604 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133938 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206994 s, 5.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (27s) == sanityn test 77f: check TBF JobID nrs policy ========== 01:36:28 (1788672988) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.376376 s, 44.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599265 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108365 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777479 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141866 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23904 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267693 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273905 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194394 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283409 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214833 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208411 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374173 s, 2.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217043 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2678 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294756 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 1+0 records in 1+0 records out [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363229 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116485 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157204 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928394 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539078 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143285 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011246 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00623445 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205344 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188647 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154598 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493061 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551744 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407777 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123239 s, 85.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242113 s, 43.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484608 s, 21.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.332818 s, 50.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447205 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672164 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102089 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162024 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980047 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160417 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22987 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246634 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234442 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245289 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222709 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196126 s, 5.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28776 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254504 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271687 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340621 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572629 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537701 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643648 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381961 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402534 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456638 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050486 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08072 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498785 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986564 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665422 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672677 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172024 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828347 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871941 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973945 s, 10.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.314486 s, 53.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495385 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498414 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706376 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924681 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108358 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866284 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125263 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287325 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835026 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111756 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102627 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356831 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212215 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778826 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117378 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102824 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269222 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725693 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838101 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671272 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446812 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459638 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525548 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533694 s, 19.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705164 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522052 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525202 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591308 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681055 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639948 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058457 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640207 s, 16.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.408448 s, 41.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371552 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807835 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657465 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180589 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215036 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220051 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183351 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15723 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229068 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188965 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212782 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222549 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189654 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19518 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229c96ec800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239243 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242245 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578253 s, 18.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667049 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185472 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105124 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836132 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699937 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201363 s, 52.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850576 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116206 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134901 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133842 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154804 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193078 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199801 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143654 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530249 s, 19.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (25s) == sanityn test 77g: Change TBF type directly ============ 01:36:53 (1788673013) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 01:37:06 (1788673026) oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 01:37:19 (1788673039) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 01:37:44 (1788673064) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.328595 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236094 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180268 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837314 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538372 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647777 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892093 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116231 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125345 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137362 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160499 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118429 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188202 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182992 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159541 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23722 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23318 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923367 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558509 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102858 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606464 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871504 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673066 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688125 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843102 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07742 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11759 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834774 s, 12.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703809 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960238 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993441 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104871 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17762 s, 5.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3083 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6998 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (76s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 01:39:00 (1788673140) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27529 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585559 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708935 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630598 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593956 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16616 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232183 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194376 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278618 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294777 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274315 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269542 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306395 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282517 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24139 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294269 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297634 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493935 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734208 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799191 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840256 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727979 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750225 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824763 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113374 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111357 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1408 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133125 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153629 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159206 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151126 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136874 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160972 s, 6.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2839 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8159 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29737 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610823 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705405 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210079 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105408 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335969 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878051 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383384 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164805 s, 63.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187961 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180933 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194181 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116736 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183836 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962792 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192855 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156901 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843858 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055066 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304286 s, 34.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087155 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320535 s, 32.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786737 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359312 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374781 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632418 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11228 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484125 s, 21.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09394 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764201 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587954 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128499 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125674 s, 8.3 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1501 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7736 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (162s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 01:41:42 (1788673302) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31426 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053649 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155254 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300897 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753993 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170995 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121759 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213096 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458536 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729641 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027908 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842578 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161428 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283904 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273862 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259819 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331973 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471267 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638393 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101979 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726248 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554474 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756777 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077308 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781076 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073581 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684979 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153091 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941608 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103597 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178331 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084118 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963864 s, 10.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0277 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.862 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33679 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748786 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932476 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858777 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339119 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17324 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105153 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420722 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080996 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218174 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122347 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146706 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546251 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590329 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197464 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253409 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174979 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969059 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705872 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620174 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674463 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108583 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753676 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109895 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103978 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116536 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127379 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106362 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867448 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086204 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137859 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113824 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116346 s, 9.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8936 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9739 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (160s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 01:44:22 (1788673462) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.578855 s, 29.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521965 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03814 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468651 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697865 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653763 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337276 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316977 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688511 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015075 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625147 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107981 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285895 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266361 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287105 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317347 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284546 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727849 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524546 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542387 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890102 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466337 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064018 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470222 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686032 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543709 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782603 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535835 s, 19.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883014 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055607 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723889 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086143 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077566 s, 13.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.64527 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97956 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.22@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.22@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.486415 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275779 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958641 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12152 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119372 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982785 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857237 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134323 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13457 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1013 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143492 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703083 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191132 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181042 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204838 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119277 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182148 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559945 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082064 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052298 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059056 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110774 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870295 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972657 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199157 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113058 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114536 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205795 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148717 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122682 s, 8.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121088 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159058 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146256 s, 7.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.57123 s, 12.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96168 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.22@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469271 s, 35.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598473 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642578 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427671 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780923 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147705 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135728 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187807 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200018 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2084 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242941 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291558 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27296 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291086 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284415 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298549 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325489 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012969 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932466 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341035 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373648 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407752 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713336 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519476 s, 20.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801877 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087843 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966897 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909827 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10197 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109359 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118747 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124837 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133856 s, 7.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.09755 s, 12.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36539 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.761127 s, 22.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615897 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146253 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627223 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834672 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965959 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196369 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184236 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164132 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16028 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226214 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135263 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114239 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131081 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154167 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18376 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16897 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169929 s, 61.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549685 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535735 s, 19.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124862 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887487 s, 11.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112463 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116437 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705583 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128046 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777914 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081129 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859407 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917203 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973038 s, 10.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082474 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984779 s, 10.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5493 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0438 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.65306 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94002 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.980649 s, 17.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358205 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844008 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559172 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13178 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227819 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163737 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212006 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206999 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230377 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248846 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242547 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213932 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212844 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240411 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226465 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337687 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047798 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615931 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523219 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544818 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676256 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817561 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644992 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540868 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621903 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494877 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06531 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585618 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951557 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919148 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904629 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975562 s, 10.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8479 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6274 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385685 s, 43.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427202 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743455 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372793 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121465 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149753 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102274 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114035 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167203 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131013 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909595 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104175 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1112 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152935 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159704 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130531 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227681 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668811 s, 15.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106582 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982209 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732313 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848749 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745462 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118395 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880245 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131193 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146173 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10222 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139948 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134567 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119435 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113591 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183691 s, 5.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73002 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98809 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.626965 s, 26.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579556 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568262 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137265 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897226 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108268 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19308 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208429 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263359 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260364 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210852 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247168 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227285 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316661 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274298 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322343 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309388 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110253 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931156 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135952 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10197 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133425 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116165 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11212 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130015 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138588 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150184 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118805 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129646 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112694 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130503 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127706 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13741 s, 7.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2021 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96698 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8221 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96653 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (456s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 01:51:58 (1788673918) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.45978 s, 36.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421362 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634464 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854136 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107175 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135553 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158901 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182393 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114567 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202521 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211947 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235628 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202531 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220557 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221682 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199721 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245893 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836331 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419145 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532503 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593177 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777846 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680294 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916407 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707009 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708453 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896604 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099591 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125275 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150519 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113205 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127302 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150938 s, 6.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.31362 s, 11.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0138 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 01:52:55 (1788673975) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.22@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.22@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.605551 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251844 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045432 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793198 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697895 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293698 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876613 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16999 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212141 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235189 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208084 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189886 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226819 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28933 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287108 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239785 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277711 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921809 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686255 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438879 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920954 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142434 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148532 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810977 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911094 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100388 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870759 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105464 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136992 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138396 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133523 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148278 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146074 s, 7.2 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.39297 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94189 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 01:53:49 (1788674029) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.22@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467231 s, 35.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396845 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495488 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136372 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165715 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175636 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167292 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151461 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240211 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241842 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21512 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213107 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190262 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215647 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274237 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226828 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266367 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06699 s, 15.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519495 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641563 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640185 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762831 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07876 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721764 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642293 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795264 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654389 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705393 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779955 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797334 s, 13.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100759 s, 10.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08519 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856875 s, 12.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.27717 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25383 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (46s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 01:54:35 (1788674075) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.538856 s, 31.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340225 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382246 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069278 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528979 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991229 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145296 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106496 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170299 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120193 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840157 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134918 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845248 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142154 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176981 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165793 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219629 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0468611 s, 22.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786644 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546591 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079863 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553993 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056198 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643406 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672424 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115584 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104408 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782799 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127304 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906944 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937787 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973637 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11471 s, 9.1 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4077 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96107 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.62888 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9496 s, 10.5 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (98s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 01:56:14 (1788674174) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.13909 s, 14.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056001 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525179 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579394 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720846 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136378 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218267 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148267 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229974 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806615 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126005 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153775 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209264 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206142 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206808 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257369 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317122 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627702 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661521 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850196 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503172 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131341 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741562 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477448 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086125 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857544 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616519 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101865 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100053 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159376 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080299 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114935 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084344 s, 12.4 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9918 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7829 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (75s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 01:57:28 (1788674248) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.593274 s, 28.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305495 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632013 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160372 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102906 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106479 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822835 s, 12.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226853 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16992 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16519 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143461 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237856 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222999 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264049 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309218 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199575 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29405 s, 3.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229c96ec800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112815 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944334 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10196 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934503 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793457 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613904 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100774 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586334 s, 17.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112662 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099414 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902323 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123581 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659268 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141685 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077714 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125183 s, 8.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.67569 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0737 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg422-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.511707 s, 32.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519622 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747286 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634223 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740897 s, 14.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449927 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500044 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250477 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868283 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692953 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183826 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130151 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242805 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373053 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390022 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35436 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394096 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744641 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438747 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362614 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615678 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385355 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00931925 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461698 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454838 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00562628 s, 186 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053529 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106209 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121578 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00965169 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010583 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161845 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025312 s, 41.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2429 s, 8.6 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0603 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4585 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0214 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (161s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 02:00:09 (1788674409) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:) Stopping client oleg422-client.virtnet /mnt/lustre opts: Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg422-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg422-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg422-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.22,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg422-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.22,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg422-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.22,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/69643/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.22,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg422-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: kmemleak disabled oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre Starting client oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre Started clients oleg422-client.virtnet: 192.168.204.122@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Started clients oleg422-client.virtnet: 192.168.204.122@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9229c96ee000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9229f2b3a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229c96ee000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229f2b3a000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.511428 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546312 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13961 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142414 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222337 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215507 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150686 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563203 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260838 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150522 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340505 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374303 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362118 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382856 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339245 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385535 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339986 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408423 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407021 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383322 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384874 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386907 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422509 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3975 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398301 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404067 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404476 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412236 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411949 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417069 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417343 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424682 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486297 s, 2.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.82956 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.61848 s, 18.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.724483 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150169 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166948 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195992 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167654 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323244 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229f2b3a000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358487 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408211 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376447 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45066 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486296 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529841 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40231 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509658 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.589905 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46141 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485115 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223319 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211363 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215097 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23225 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203868 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23448 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241074 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26669 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269729 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24821 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25354 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284076 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278875 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298277 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303241 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386366 s, 2.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9318 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96888 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (200s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 02:03:30 (1788674610) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 02:03:47 (1788674627) On MGS 192.168.204.122, active = nodemap.active=1 On MGS 192.168.204.122, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662796 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966299 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103177 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189931 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220811 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171358 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257586 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292411 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291257 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295759 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24754 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296695 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321621 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264239 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229413 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269882 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328441 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778376 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781206 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994895 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922491 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934431 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672272 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659485 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824352 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807139 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084226 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110691 s, 9.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841314 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150804 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996748 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101105 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11245 s, 9.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.87177 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.43996 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.122, TBF.id = On MGS 192.168.204.122, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.740754 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718426 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657818 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746755 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127161 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19245 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138205 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113996 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255039 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137626 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286475 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362407 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177389 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229f2b3a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270568 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298004 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367421 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468745 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199596 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295483 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477702 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117279 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107355 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656411 s, 16.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368007 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815562 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128039 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881966 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745122 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926188 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13781 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759686 s, 13.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107983 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712805 s, 14.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.65851 s, 13.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.12469 s, 33.6 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg422-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 On MGS 192.168.204.122, active = nodemap.active=1 On MGS 192.168.204.122, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.635939 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583899 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785437 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104095 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135849 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128735 s, 8.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175861 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150163 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113252 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244722 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224915 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283651 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266629 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260775 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232326 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288188 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316577 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229f2b3a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191796 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172496 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158786 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181259 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183507 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175978 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17322 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193489 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195947 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210214 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210138 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272559 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208861 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208615 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219821 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225215 s, 4.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg422-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg422-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.45958 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9443 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.122, TBF.id = On MGS 192.168.204.122, active = nodemap.active=0 PASS 77kj (179s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 02:06:47 (1788674807) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 02:06:59 (1788674819) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg422-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.57778 s, 293 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg422-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.007 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 02:07:58 (1788674878) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.645112 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134628 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125065 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170295 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16662 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226093 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14699 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17865 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260925 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238343 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238592 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227229 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236757 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177894 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14525 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378681 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242491 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319496 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416955 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783052 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457309 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476953 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495999 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460565 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629284 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683661 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907769 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678967 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119337 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717092 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843562 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759988 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116976 s, 9.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.90906 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.4762 s, 19.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.584703 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575426 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118705 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863438 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11592 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209203 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2178 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155628 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201588 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211757 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191908 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226746 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284281 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213565 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269965 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268302 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31997 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9229f2b3a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9229f2b3a000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157496 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149107 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17875 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191151 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196009 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147683 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206421 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20259 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185717 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171267 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197014 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177399 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202472 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178322 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178023 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183518 s, 5.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg422-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg422-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.6168 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.10687 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 02:09:26 (1788674966) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (14s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 02:09:40 (1788674980) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (128s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 02:11:48 (1788675108) mds.MDS.mdt.nrs_policies=tbf oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg422-client: oleg422-server: ssh exited with exit code 36 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg422-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg422-client: oleg422-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (37s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 02:12:25 (1788675145) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273937 kill md_thread xargs PASS 77r (54s) == sanityn test 77s: Check TBF LRU shrinker ============== 02:13:19 (1788675199) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (67s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 02:14:26 (1788675266) oleg422-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg422-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg422-client: oleg422-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 02:14:37 (1788675277) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 02:14:39.000000000 -0400 Modify: 2026-09-06 02:14:39.000000000 -0400 Change: 2026-09-06 02:14:39.000000000 -0400 Birth: 2026-09-06 02:14:39.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (18s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 02:14:55 (1788675295) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 02:14:59 (1788675299) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 02:15:03 (1788675303) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 02:15:06 (1788675306) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 02:15:09 (1788675309) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 02:15:12 (1788675312) - open/close 670 (time 1788675327.28 total 10.01 last 66.96) - open/close 1379 (time 1788675337.28 total 20.01 last 70.90) - open/close 2049 (time 1788675347.29 total 30.02 last 66.94) total: 2500 open/close in 36.41 seconds: 68.66 ops/second snapshot_time 1788675443.282971494 secs.nsecs start_time 1788675315.041133717 secs.nsecs elapsed_time 128.241837777 secs.nsecs open 2505 samples [usecs] 128 24714 4430051 12493521017 close 2505 samples [usecs] 191 8982 1168421 1260673683 mknod 2501 samples [usecs] 541 24597 3829504 9640901460 unlink 2500 samples [usecs] 777 106187 7209158 102854470130 mkdir 2 samples [usecs] 982 1541 2523 3339005 rmdir 2 samples [usecs] 9551 17852 27403 409915505 rename 2500 samples [usecs] 570 44207 17802955 240343685113 getattr 3760 samples [usecs] 8 15136 305810 803956834 parallel_rename_file 2500 samples [usecs] 570 44207 17802955 240343685113 crossdir_rename 2500 samples [usecs] 570 44207 17802955 240343685113 readpage 32 samples [usecs] 5137 86003 943310 39343410468 PASS 81d (135s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 02:17:27 (1788675447) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7620 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 02:17:34 (1788675454) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (2s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 02:17:37 (1788675457) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 02:17:48 (1788675468) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1788675470 Finishing test test1 at 1788675474 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 02:17:58 (1788675478) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 02:18:00 (1788675480) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 02:18:03 (1788675483) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 02:18:06 (1788675486) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5800 0x16a8 0x240000400 1 5763 0x1683 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5801 0x16a9 0x240000400 1 5764 0x1684 0x280000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 02:18:23 (1788675503) osc.lustre-OST0000-osc-ffff9229c96ee000.idle_timeout=0 osc.lustre-OST0000-osc-ffff9229f2b3a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9229c96ee000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9229f2b3a000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.164396 s, 49.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98081 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290594 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0921637 s, 44.4 kB/s osc.lustre-OST0000-osc-ffff9229c96ee000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9229f2b3a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229c96ee000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9229f2b3a000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 02:18:39 (1788675519) llite.lustre-ffff9229c96ee000.fast_read=0 llite.lustre-ffff9229f2b3a000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0699793 s, 30.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9229c96ee000.fast_read=1 llite.lustre-ffff9229f2b3a000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 02:18:59 (1788675539) llite.lustre-ffff9229c96ee000.fast_read=0 llite.lustre-ffff9229f2b3a000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0134757 s, 912 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9229c96ee000.fast_read=1 llite.lustre-ffff9229f2b3a000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 02:19:12 (1788675552) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 02:19:15 (1788675555) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00539529 s, 759 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 02:19:22 (1788675562) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159464 s, 13.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 02:19:30 (1788675570) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.248591 s, 8.4 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 02:19:37 (1788675577) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.019972 s, 601 kB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 02:19:45 (1788675585) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00787247 s, 520 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 02:19:52 (1788675592) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0346935 s, 118 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 02:19:59 (1788675599) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00730521 s, 561 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 02:20:08 (1788675608) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 02:20:16 (1788675616) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1788675618 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1788675622 Finishing test test23 at 1788675622 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 02:20:29 (1788675629) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 02:20:33 (1788675633) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 02:20:56 (1788675656) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 02:20:59 (1788675659) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.018604 s, 55.0 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 02:21:01.000000000 -0400 Modify: 2026-09-06 02:21:01.000000000 -0400 Change: 2026-09-06 02:21:01.000000000 -0400 Birth: 2026-09-06 02:21:01.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1788675661 1788675661 1788675661 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 02:21:01.000000000 -0400 Modify: 2026-09-06 02:21:01.000000000 -0400 Change: 2026-09-06 02:21:01.000000000 -0400 Birth: 2026-09-06 02:21:01.000000000 -0400 1788675661 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 02:21:07 (1788675667) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 02:21:15 (1788675675) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.342959 s, 30.6 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5810 0x16b2 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 02:21:25 (1788675685) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333024 s, 31.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5771 0x168b 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7620 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 02:21:39 (1788675699) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0115796 s, 864 kB/s llite.lustre-ffff9229c96ee000.hybrid_io=0 llite.lustre-ffff9229f2b3a000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.5467 s, 1.8 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 02:21:51 (1788675711) 192.168.204.122@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre 192.168.204.122@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: kmemleak disabled oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 192.168.204.122@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg422-client.virtnet /mnt/lustre (opts:) Stopping client oleg422-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: kmemleak disabled oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: MODOPTS_LIBCFS= oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre2 PASS 109 (67s) == sanityn test 110: do not grant another lock on resend ========================================================== 02:22:58 (1788675778) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 02:23:00.000000000 -0400 Modify: 2026-09-06 02:23:00.000000000 -0400 Change: 2026-09-06 02:23:00.000000000 -0400 Birth: 2026-09-06 02:23:00.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg422-client.virtnet oleg422-client.virtnet PASS 110 (168s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 02:25:46 (1788675946) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 02:25:50 (1788675950) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 02:25:53 (1788675953) lustre-OST0000-osc-ffff9229e4f70800 active. lustre-OST0001-osc-ffff9229e4f70800 active. lustre-OST0000-osc-ffff9229e4f70800 active. lustre-OST0001-osc-ffff9229e4f70800 active. MGC192.168.204.122@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 02:25:59 (1788675959) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 02:26:04 (1788675964) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 02:26:08 (1788675968) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 02:26:12 (1788675972) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.012989 s, 1.6 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.109249 s, 187 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 02:26:23 (1788675983) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 02:26:33 (1788675993) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1759 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2305 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2135 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.194 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.164 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2483 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.22 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0842 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.21 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2233 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2179 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2226 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2626 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2587 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2376 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2398 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4561 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3826 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg422-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9229c53f5000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9229c53f5000.ost_server_uuid in IDLE state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 10534 sec ============= 02:28:00 (1788676080) === sanityn: start cleanup 02:28:02 (1788676082) === Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg422-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 02:33:16 (1788676396) ===