-----============= acceptance-small: sanityn ============----- Tue Sep 22 14:05:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITYN_ONLY='' SANITYN_EXCEPT='' oleg435-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 56b skipping tests SLOW=no: 33a === sanityn: start setup 14:05:15 (1790100315) === Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Mount client oleg435-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c6482e71800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c64852b4800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c6482e71800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c64852b4800.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:05:37 (1790100337) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181566 s, 57.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7630] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:05:38 (1790100338) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:05:43 (1790100343) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (4s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:05:47 (1790100347) -rw-r--r-- 1 root root 0 Sep 22 14:05 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:05:51 (1790100351) -rw-r--r-- 1 root root 0 Sep 22 14:05 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:05:55 (1790100355) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (4s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:05:59 (1790100359) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:06:03 (1790100363) total 1 -rwxrwxrwx 1 root root 0 Sep 22 14:05 f2a -rwxrwxrwx 1 root root 0 Sep 22 14:05 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 22 14:05 f2a -rwxrwxrwx 1 root root 0 Sep 22 14:05 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0110251 s, 46.4 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:06:07 (1790100367) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 14:06:09 (1790100369) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27944 s, 7.5 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:06:13 (1790100373) PASS 3 (4s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:06:17 (1790100377) PASS 4 (5s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:06:22 (1790100382) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 14:06:27 (1790100387) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (4s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:06:31 (1790100391) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:06:35 (1790100395) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:06:39 (1790100399) PASS 9a (4s) == sanityn test 9b: append to striped sparse file ======== 14:06:43 (1790100403) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298035 s, 35.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0372754 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:06:47 (1790100407) 1+0 records in 1+0 records out 1 byte copied, 0.0105642 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0197206 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0164085 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118845 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00943073 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0143737 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00832288 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0159651 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132306 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0122404 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.011894 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114769 s, 0.1 kB/s PASS 10a (5s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:06:52 (1790100412) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0109126 s, 282 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0241008 s, 170 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000323803 s, 9.5 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:06:55 (1790100415) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 304: /mnt/lustre2/d11/f: Text file busy PASS 11 (4s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:06:59 (1790100419) warning: '-runtime' deprecated, use '-t runtime' instead - link 527 (time 1790100430.87 total 10.04 last 52.49) - link 856 (time 1790100440.88 total 20.05 last 32.87) - link 1099 (time 1790100450.88 total 30.05 last 24.29) - link 1309 (time 1790100460.91 total 40.08 last 20.94) - link 1495 (time 1790100470.93 total 50.11 last 18.56) - link 1676 (time 1790100480.97 total 60.14 last 18.04) - link 1859 (time 1790100490.97 total 70.14 last 18.30) - link 2046 (time 1790100500.98 total 80.15 last 18.67) - link 2280 (time 1790100511.04 total 90.21 last 23.26) total: 2506 link in 100.03 seconds: 25.05 ops/second using seed 4196595737 running for 100 seconds - stat 6942 (time 1790100432 ; total 11 ; last 11) - stat 10000 (time 1790100436 ; total 15 ; last 4) - stat 15169 (time 1790100447 ; total 26 ; last 11) - stat 19201 (time 1790100458 ; total 37 ; last 11) - stat 20000 (time 1790100462 ; total 41 ; last 4) - stat 22370 (time 1790100473 ; total 52 ; last 11) - stat 23602 (time 1790100484 ; total 63 ; last 11) - stat 24992 (time 1790100495 ; total 74 ; last 11) - stat 27082 (time 1790100506 ; total 85 ; last 11) - stat 28608 (time 1790100517 ; total 96 ; last 11) total: 29425 stats in 100 seconds: 294.250000 stats/second - unlinked 0 (time 1790100471 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2506) error: No such file or directory total: 2505 unlinks in 91 seconds: 27.527473 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21428) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21431) - No such process PASS 12 (150s) == sanityn test 13: test directory page revocation ======= 14:09:30 (1790100570) f13 f13 PASS 13 (10s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:09:41 (1790100581) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 339: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 22618 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (9s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:09:49 (1790100589) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 23197 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:09:58 (1790100598) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 354: 23775 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:10:05 (1790100605) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 366: 24358 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:10:13 (1790100613) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 377: 24940 Terminated $DIR1/$tdir/sleep 60 PASS 14d (10s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:10:23 (1790100623) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (55s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:11:18 (1790100678) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.538988 s, 15.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.373864 s, 22.4 MB/s Chance of close/open is 1 in 50 Seed set to 683 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x70273d 000001[0] 1790100684.158902 trunc from 00000000 to 0x70273c (0x70273d bytes) 000005[1] 1790100684.299106 punch from 0x08ba28 to 0x090832 (0x4e0b bytes) 000006[0] 1790100684.361060 mapread 0x044758 thru 0x05142e (0xccd7 bytes) 000008[0] 1790100684.579369 mapwrite 0x73a2db thru 0x74a158 (0xfe7e bytes) 000009[0] 1790100684.839469 read 0x005e81 thru 0x00a867 (0x49e7 bytes) 000010[1] 1790100684.859557 mapwrite 0x950c78 thru 0x95e6ba (0xda43 bytes) 000011[1] 1790100685.087625 trunc from 0x95e6bb to 0x20015e (0x75e55c bytes) 000012[1] 1790100685.125945 mapwrite 0x3c2908 thru 0x3c3bd7 (0x12d0 bytes) 000014[0] 1790100685.269808 mapread 0x00ff1a thru 0x01fb54 (0xfc3b bytes) 000016[1] 1790100685.356774 read 0x3140b8 thru 0x315d55 (0x1c9e bytes) 000021[1] 1790100685.413118 read 0x1ae3e4 thru 0x1b1028 (0x2c45 bytes) 000025[1] 1790100685.559435 write 0x069115 thru 0x0774cb (0xe3b7 bytes) 000030[1] 1790100685.684346 read 0x1d22e0 thru 0x1d937f (0x70a0 bytes) 000031[0] 1790100685.786038 write 0x563d6e thru 0x566cec (0x2f7f bytes) 000033[0] 1790100685.830916 mapread 0x369320 thru 0x37727b (0xdf5c bytes) 000034[0] 1790100685.916659 write 0x4a883c thru 0x4adb33 (0x52f8 bytes) 000037[0] 1790100686.037245 read 0x510a83 thru 0x5110d0 (0x64e bytes) 000039[0] 1790100686.054641 mapread 0x2d3255 thru 0x2e2cea (0xfa96 bytes) 000041[0] 1790100686.132321 punch from 0x35f4b9 to 0x363081 (0x3bc9 bytes) 000043[1] 1790100686.200157 write 0x14d3e2 thru 0x158adf (0xb6fe bytes) 000044[1] 1790100686.246510 punch from 0x1f26f2 to 0x1fc643 (0x9f52 bytes) 000047[1] 1790100686.319962 trunc from 0x566ced to 0x57e7b6 (0x17aca bytes) 000048[1] 1790100686.451914 read 0x0fda0d thru 0x10188d (0x3e81 bytes) 000049[1] 1790100686.545653 mapwrite 0x5c7c5d thru 0x5d72ad (0xf651 bytes) 000051[1] 1790100686.862168 mapwrite 0x97c1b7 thru 0x9829fd (0x6847 bytes) 000053[0] 1790100687.107098 write 0x08de08 thru 0x09c745 (0xe93e bytes) 000054[0] 1790100687.143344 mapwrite 0x09f7f3 thru 0x0aa3aa (0xabb8 bytes) 000055[0] 1790100687.254604 punch from 0x2e5617 to 0x2f1834 (0xc21e bytes) 000058[1] 1790100687.315429 mapwrite 0x0ba9b8 thru 0x0c9c41 (0xf28a bytes) 000059[1] 1790100687.532575 punch from 0x0e87ad to 0x0ee3a1 (0x5bf5 bytes) 000060[1] 1790100687.576607 mapread 0x68d0fc thru 0x68f860 (0x2765 bytes) 000064[1] 1790100687.679352 mapread 0x540a76 thru 0x54d65e (0xcbe9 bytes) 000066[0] 1790100687.744633 trunc from 0x9829fe to 0x249aaa (0x738f53 bytes) 000067[1] 1790100687.792146 punch from 0x21a097 to 0x21ddc5 (0x3d2f bytes) 000068[0] 1790100687.826469 trunc from 0x249aab to 0x4f9bce (0x2b0124 bytes) 000069[0] 1790100687.864156 punch from 0x445b61 to 0x4499e9 (0x3e89 bytes) 000072[1] 1790100687.937397 mapwrite 0x77b846 thru 0x782f22 (0x76dd bytes) 000074[0] 1790100688.150461 write 0x4e3fac thru 0x4e450c (0x561 bytes) 000076[0] 1790100688.211388 punch from 0x11e497 to 0x12355a (0x50c4 bytes) 000077[0] 1790100688.247784 punch from 0x5f318d to 0x5f6aef (0x3963 bytes) 000078[1] 1790100688.276932 write 0x068042 thru 0x06db4c (0x5b0b bytes) 000079[0] 1790100688.371344 write 0x5bb44f thru 0x5c4229 (0x8ddb bytes) 000080[1] 1790100688.444247 mapwrite 0x9d6c48 thru 0x9e1994 (0xad4d bytes) truncating to largest ever: 0x82a37e 000083[1] 1790100688.801323 trunc from 0x9e1995 to 0x82a37d (0x1b7617 bytes) truncating to largest ever: 0x853fc5 000084[1] 1790100688.853999 trunc from 0x82a37e to 0x853fc4 (0x29c47 bytes) 000085[0] 1790100688.875014 read 0x61746a thru 0x61b0bf (0x3c56 bytes) 000086[1] 1790100688.928890 mapwrite 0x1a4d8b thru 0x1b3b11 (0xed87 bytes) 000087[0] 1790100689.133124 mapread 0x82b3da thru 0x82d3ce (0x1ff5 bytes) 000089[0] 1790100689.184393 write 0x97e6fc thru 0x98a966 (0xc26b bytes) 000090[1] 1790100689.209495 mapwrite 0x37da58 thru 0x38b9c3 (0xdf6c bytes) 000093[1] 1790100689.414167 mapread 0x66516b thru 0x66e190 (0x9026 bytes) 000094[1] 1790100689.481332 read 0x83207a thru 0x837539 (0x54c0 bytes) 000095[1] 1790100689.513512 read 0x5cde4f thru 0x5db9c1 (0xdb73 bytes) 000096[0] 1790100689.543738 punch from 0x4d88a1 to 0x4e41b1 (0xb911 bytes) 000098[1] 1790100689.606979 mapwrite 0x089ba4 thru 0x09246a (0x88c7 bytes) 000099[0] 1790100689.724931 mapwrite 0x8362df thru 0x83d48f (0x71b1 bytes) 000100[0] 1790100689.885535 write 0x06517d thru 0x072006 (0xce8a bytes) 000101[1] 1790100689.911790 read 0x7d6b54 thru 0x7e6651 (0xfafe bytes) 000103[1] 1790100689.971118 trunc from 0x98a967 to 0x4195ec (0x57137a bytes) 000106[1] 1790100690.045063 mapread 0x3a4eee thru 0x3b42eb (0xf3fe bytes) 000107[0] 1790100690.115226 trunc from 0x4195ed to 0x3539f9 (0xc5bf3 bytes) 000108[1] 1790100690.214600 read 0x1db7c4 thru 0x1e9769 (0xdfa6 bytes) 000110[0] 1790100690.272039 trunc from 0x3539fa to 0x25a99b (0xf905e bytes) 000111[0] 1790100690.354867 mapwrite 0x6d568a thru 0x6e0887 (0xb1fe bytes) 000112[1] 1790100690.490922 write 0x01054e thru 0x01629d (0x5d50 bytes) 000113[1] 1790100690.560217 trunc from 0x6e0888 to 0x1cade5 (0x515aa2 bytes) 000114[0] 1790100690.617628 trunc from 0x1cade6 to 0x1497bf (0x81626 bytes) 000118[0] 1790100690.735689 mapread 0x0d6d76 thru 0x0dfac8 (0x8d53 bytes) 000120[0] 1790100690.812665 write 0x2e3e89 thru 0x2e8669 (0x47e1 bytes) 000121[1] 1790100690.814851 read 0x03136c thru 0x03e3f7 (0xd08c bytes) 000126[1] 1790100690.939101 punch from 0x000295 to 0x009df5 (0x9b61 bytes) 000127[1] 1790100690.987698 read 0x1d41bc thru 0x1dafc1 (0x6e06 bytes) 000128[1] 1790100691.090489 read 0x01029e thru 0x019571 (0x92d4 bytes) 000130[0] 1790100691.140821 mapwrite 0x6369f5 thru 0x640526 (0x9b32 bytes) 000132[1] 1790100691.341056 mapwrite 0x2693bd thru 0x272006 (0x8c4a bytes) 000133[1] 1790100691.502887 read 0x056a07 thru 0x05f653 (0x8c4d bytes) 000134[0] 1790100691.520056 trunc from 0x640527 to 0x7bb901 (0x17b3db bytes) 000135[0] 1790100691.573670 mapread 0x1b6cb1 thru 0x1b9b18 (0x2e68 bytes) 000138[0] 1790100691.739570 trunc from 0x7bb902 to 0x4caa41 (0x2f0ec0 bytes) 000139[0] 1790100691.806637 write 0x2f79c6 thru 0x2fb9bd (0x3ff8 bytes) 000143[0] 1790100691.841919 mapwrite 0x2022bc thru 0x211a20 (0xf765 bytes) truncating to largest ever: 0x8cb842 000144[0] 1790100691.970590 trunc from 0x4caa42 to 0x8cb841 (0x400e00 bytes) 000145[1] 1790100692.059639 read 0x2a89eb thru 0x2af3d4 (0x69ea bytes) 000147[0] 1790100692.155093 mapread 0x1cd53e thru 0x1d7f6a (0xaa2d bytes) 000148[0] 1790100692.205155 mapread 0x319e96 thru 0x3214a3 (0x760e bytes) 000149[1] 1790100692.248053 punch from 0x4d2259 to 0x4dd0d4 (0xae7c bytes) 000150[1] 1790100692.313641 mapwrite 0x809509 thru 0x8147db (0xb2d3 bytes) 000151[1] 1790100692.546906 mapread 0x30a8e9 thru 0x30d499 (0x2bb1 bytes) 000152[0] 1790100692.607153 trunc from 0x8cb842 to 0x77b890 (0x14ffb1 bytes) 000154[0] 1790100692.699867 mapwrite 0x06aff8 thru 0x0760cb (0xb0d4 bytes) 000155[0] 1790100692.818163 write 0x2cfa31 thru 0x2d64ce (0x6a9e bytes) 000157[1] 1790100692.872395 write 0x91e222 thru 0x9227a7 (0x4586 bytes) 000159[1] 1790100692.929593 mapwrite 0x933688 thru 0x93a0a9 (0x6a22 bytes) 000160[1] 1790100693.112018 write 0x5a8493 thru 0x5ab377 (0x2ee5 bytes) 000162[0] 1790100693.133741 write 0x7a048f thru 0x7a4a6d (0x45df bytes) 000165[1] 1790100693.243661 read 0x2a6d95 thru 0x2b4d40 (0xdfac bytes) 000167[1] 1790100693.288652 mapread 0x3e2a94 thru 0x3e9c27 (0x7194 bytes) 000168[0] 1790100693.331949 mapwrite 0x904901 thru 0x91447a (0xfb7a bytes) 000169[0] 1790100693.500912 trunc from 0x93a0aa to 0x4ed078 (0x44d031 bytes) 000170[1] 1790100693.531523 mapwrite 0x02a59e thru 0x0397b6 (0xf219 bytes) 000173[0] 1790100693.782517 read 0x0753f0 thru 0x07e825 (0x9436 bytes) 000175[1] 1790100693.841571 punch from 0x13e94c to 0x143c0e (0x52c3 bytes) 000176[1] 1790100693.879100 write 0x85507b thru 0x856145 (0x10cb bytes) 000177[0] 1790100693.896554 read 0x340fbe thru 0x3448f2 (0x3935 bytes) 000180[0] 1790100693.948536 punch from 0x4601bb to 0x46f34b (0xf191 bytes) 000183[1] 1790100693.985433 trunc from 0x856146 to 0x4362a3 (0x41fea2 bytes) 000184[1] 1790100694.125815 trunc from 0x4362a4 to 0x30fce1 (0x1265c2 bytes) 000187[1] 1790100694.265012 read 0x0c0bcc thru 0x0cbc57 (0xb08c bytes) 000188[1] 1790100694.377824 mapread 0x03599b thru 0x0417e3 (0xbe49 bytes) 000191[1] 1790100694.435927 mapread 0x03f3ee thru 0x04c49a (0xd0ad bytes) 000193[0] 1790100694.441831 punch from 0x0e75d0 to 0x0f319c (0xbbcd bytes) 000194[0] 1790100694.486179 trunc from 0x30fce2 to 0x87660f (0x56692e bytes) 000195[0] 1790100694.539082 punch from 0x39c963 to 0x3a26d1 (0x5d6f bytes) 000197[0] 1790100694.583069 trunc from 0x876610 to 0x65ac53 (0x21b9bc bytes) 000199[0] 1790100694.643203 punch from 0x546bf3 to 0x550666 (0x9a74 bytes) 000202[1] 1790100694.756149 write 0x44d193 thru 0x459a2c (0xc89a bytes) 000204[0] 1790100694.849650 write 0x0a7ae7 thru 0x0b6381 (0xe89b bytes) 000206[1] 1790100694.898712 punch from 0x5ae1b2 to 0x5bd48e (0xf2dd bytes) 000208[1] 1790100694.932187 write 0x3ee8ff thru 0x3f0ca5 (0x23a7 bytes) 000209[1] 1790100694.967976 punch from 0x4b6eb6 to 0x4c5c9e (0xede9 bytes) 000210[1] 1790100695.005897 mapread 0x466794 thru 0x46893a (0x21a7 bytes) 000212[1] 1790100695.068385 read 0x5f8d16 thru 0x6061bb (0xd4a6 bytes) 000213[0] 1790100695.149218 mapwrite 0x638342 thru 0x63912d (0xdec bytes) 000214[1] 1790100695.285713 punch from 0x49aac9 to 0x4a89f0 (0xdf28 bytes) 000215[1] 1790100695.339323 mapwrite 0x64d947 thru 0x657a4a (0xa104 bytes) 000216[1] 1790100695.424077 trunc from 0x65ac54 to 0x70a8fc (0xafca9 bytes) 000217[1] 1790100695.484510 trunc from 0x70a8fd to 0x141d84 (0x5c8b78 bytes) 000220[1] 1790100695.520034 mapread 0x062580 thru 0x07225a (0xfcdb bytes) 000221[0] 1790100695.602125 mapread 0x0cd708 thru 0x0ce3e2 (0xcdb bytes) 000222[0] 1790100695.656564 mapwrite 0x6c4ee4 thru 0x6c56ad (0x7ca bytes) 000223[0] 1790100695.827092 punch from 0x386665 to 0x38a165 (0x3b01 bytes) 000224[0] 1790100695.867141 punch from 0x58b88a to 0x594a17 (0x918e bytes) 000227[1] 1790100695.899797 write 0x59043c thru 0x59d986 (0xd54b bytes) 000228[1] 1790100695.944915 write 0x3e7c97 thru 0x3ec592 (0x48fc bytes) 000232[0] 1790100696.038028 trunc from 0x6c56ae to 0x3865c4 (0x33f0e9 bytes) 000233[0] 1790100696.106835 mapread 0x1fe076 thru 0x207baa (0x9b35 bytes) 000236[1] 1790100696.154569 punch from 0x1ee138 to 0x1fd0ef (0xefb8 bytes) 000238[0] 1790100696.227147 trunc from 0x3865c5 to 0x692c03 (0x30c63f bytes) 000239[0] 1790100696.278221 read 0x1f00d2 thru 0x1f1201 (0x1130 bytes) 000240[1] 1790100696.347935 mapread 0x660f46 thru 0x670c3a (0xfcf5 bytes) 000242[0] 1790100696.560908 write 0x494a16 thru 0x4a34fc (0xeae7 bytes) 000247[0] 1790100696.740296 read 0x4967f6 thru 0x4a3f45 (0xd750 bytes) 000248[0] 1790100696.803217 write 0x731a86 thru 0x737a43 (0x5fbe bytes) 000249[1] 1790100696.844776 write 0x42c4f5 thru 0x439c4d (0xd759 bytes) 000251[0] 1790100696.968672 read 0x1a7038 thru 0x1b4e3e (0xde07 bytes) 000252[0] 1790100697.011968 punch from 0x52b666 to 0x52ccfc (0x1697 bytes) 000253[0] 1790100697.088862 write 0x5aba5b thru 0x5aee2d (0x33d3 bytes) 000255[1] 1790100697.144693 punch from 0x1d1f1f to 0x1d7337 (0x5419 bytes) 000256[0] 1790100697.187712 trunc from 0x737a44 to 0x19ad4c (0x59ccf7 bytes) 000257[1] 1790100697.260425 trunc from 0x19ad4d to 0x80ea79 (0x673d2d bytes) 000258[1] 1790100697.337385 mapread 0x01b5f1 thru 0x01b979 (0x389 bytes) 000263[0] 1790100697.419553 punch from 0x6d624d to 0x6e57a5 (0xf559 bytes) 000265[0] 1790100697.546028 trunc from 0x80ea7a to 0x68d567 (0x181512 bytes) 000267[0] 1790100697.649008 mapread 0x6328bd thru 0x642314 (0xfa58 bytes) 000272[0] 1790100697.850085 mapread 0x52817f thru 0x534e66 (0xcce8 bytes) 000273[0] 1790100697.904254 write 0x8016ae thru 0x80af44 (0x9897 bytes) 000277[0] 1790100698.016152 read 0x27c3db thru 0x2815bc (0x51e2 bytes) 000279[1] 1790100698.062945 write 0x1ed4cb thru 0x1f67b3 (0x92e9 bytes) 000280[0] 1790100698.131895 write 0x6637ca thru 0x66fc78 (0xc4af bytes) 000283[1] 1790100698.211559 read 0x7c3de3 thru 0x7ccbbc (0x8dda bytes) 000286[1] 1790100698.361739 trunc from 0x80af45 to 0x816613 (0xb6cf bytes) 000288[1] 1790100698.422969 trunc from 0x816614 to 0x1a54ab (0x671168 bytes) 000289[1] 1790100698.493374 mapread 0x0631c3 thru 0x072b5b (0xf999 bytes) 000293[1] 1790100698.581233 read 0x15a45a thru 0x161ff3 (0x7b9a bytes) 000294[1] 1790100698.665973 read 0x120808 thru 0x125ee3 (0x56dc bytes) 000295[0] 1790100698.728746 punch from 0x1a3a34 to 0x1a54ab (0x1a78 bytes) 000297[0] 1790100698.793862 trunc from 0x1a54ac to 0x237f30 (0x92a85 bytes) 000299[1] 1790100698.870997 read 0x075496 thru 0x081d3d (0xc8a8 bytes) 000300[0] 1790100698.981167 read 0x174a0a thru 0x181678 (0xcc6f bytes) 000303[1] 1790100699.142587 punch from 0x1ade5a to 0x1ba887 (0xca2e bytes) 000304[0] 1790100699.228378 trunc from 0x237f31 to 0x08ecb6 (0x1a927a bytes) 000306[0] 1790100699.328023 punch from 0x01e2de to 0x02a8b5 (0xc5d8 bytes) 000307[0] 1790100699.366686 write 0x50b8a8 thru 0x515dd1 (0xa52a bytes) 000308[0] 1790100699.374248 punch from 0x185742 to 0x187144 (0x1a03 bytes) 000311[0] 1790100699.382541 mapwrite 0x948225 thru 0x94f6f7 (0x74d3 bytes) 000312[0] 1790100699.787059 mapread 0x75ce1e thru 0x76b6c8 (0xe8ab bytes) truncating to largest ever: 0x9aea44 000316[0] 1790100699.825202 trunc from 0x94f6f8 to 0x9aea43 (0x5f34c bytes) 000317[0] 1790100699.856185 trunc from 0x9aea44 to 0x24b7c3 (0x763280 bytes) 000318[1] 1790100699.924042 mapread 0x168514 thru 0x175060 (0xcb4d bytes) 000319[0] 1790100700.141642 mapwrite 0x1b7181 thru 0x1b88ad (0x172d bytes) 000325[0] 1790100700.497169 mapwrite 0x18301e thru 0x1832a4 (0x287 bytes) 000326[0] 1790100700.569225 punch from 0x21aa11 to 0x2284a8 (0xda98 bytes) 000327[0] 1790100700.602672 read 0x1cafae thru 0x1d4efd (0x9f50 bytes) 000329[0] 1790100700.712438 mapread 0x101ea9 thru 0x10f778 (0xd8d0 bytes) 000330[1] 1790100700.715687 mapread 0x23f095 thru 0x24b7c3 (0xc72f bytes) 000331[0] 1790100700.938093 mapwrite 0x3ff38a thru 0x4080af (0x8d26 bytes) 000332[1] 1790100701.170519 mapread 0x02069a thru 0x0241d0 (0x3b37 bytes) 000333[1] 1790100701.296191 punch from 0x3de21c to 0x3e24f6 (0x42db bytes) 000335[1] 1790100701.393392 mapwrite 0x1a0337 thru 0x1a9fa8 (0x9c72 bytes) 000336[1] 1790100701.651407 mapread 0x2c4d05 thru 0x2d30a9 (0xe3a5 bytes) 000337[1] 1790100701.805070 trunc from 0x4080b0 to 0x4d8d24 (0xd0c75 bytes) 000338[1] 1790100701.883239 read 0x4b64bd thru 0x4c3240 (0xcd84 bytes) 000340[0] 1790100701.980182 mapread 0x02cda2 thru 0x037d53 (0xafb2 bytes) 000341[1] 1790100702.116153 mapread 0x1ee4fb thru 0x1f5804 (0x730a bytes) 000342[1] 1790100702.169038 punch from 0x474d1e to 0x481660 (0xc943 bytes) 000343[0] 1790100702.175355 punch from 0x037708 to 0x0385f3 (0xeec bytes) 000345[1] 1790100702.304454 read 0x385a87 thru 0x393d91 (0xe30b bytes) 000347[1] 1790100702.387638 punch from 0x3ed4ca to 0x3ee63b (0x1172 bytes) 000348[1] 1790100702.434376 read 0x2bdaf8 thru 0x2ccfc4 (0xf4cd bytes) 000349[0] 1790100702.512888 trunc from 0x4d8d25 to 0x7bef6c (0x2e6248 bytes) 000350[1] 1790100702.625630 write 0x35c9a9 thru 0x35e993 (0x1feb bytes) 000351[0] 1790100702.680801 write 0x97ffd9 thru 0x988a5d (0x8a85 bytes) 000356[0] 1790100702.742474 write 0x01d708 thru 0x026fe0 (0x98d9 bytes) 000358[1] 1790100702.754430 read 0x620066 thru 0x620434 (0x3cf bytes) 000359[0] 1790100702.796548 mapwrite 0x86babe thru 0x87b03f (0xf582 bytes) 000360[0] 1790100703.031001 mapwrite 0x8fd087 thru 0x90156c (0x44e6 bytes) 000363[0] 1790100703.201023 read 0x78ec3d thru 0x7956b0 (0x6a74 bytes) 000365[1] 1790100703.241130 punch from 0x57ca42 to 0x58512d (0x86ec bytes) 000367[0] 1790100703.309695 punch from 0x48896e to 0x488c34 (0x2c7 bytes) 000368[1] 1790100703.352972 mapwrite 0x859dd2 thru 0x8628e2 (0x8b11 bytes) 000370[1] 1790100703.543202 mapread 0x8c34c3 thru 0x8c9899 (0x63d7 bytes) 000371[0] 1790100703.590663 read 0x24539f thru 0x24cba4 (0x7806 bytes) 000373[1] 1790100703.712753 write 0x9cc1e7 thru 0x9d2785 (0x659f bytes) 000374[0] 1790100703.765792 write 0x7a8fb7 thru 0x7acf9f (0x3fe9 bytes) 000377[0] 1790100703.906685 punch from 0x986c5d to 0x98bc17 (0x4fbb bytes) 000378[0] 1790100703.999955 trunc from 0x9d2786 to 0x6147ed (0x3bdf98 bytes) 000380[1] 1790100704.066648 punch from 0x183cc6 to 0x187f87 (0x42c2 bytes) 000381[0] 1790100704.114423 mapwrite 0x1ba576 thru 0x1bf01f (0x4aaa bytes) 000382[0] 1790100704.233155 write 0x70fc53 thru 0x71b799 (0xbb47 bytes) 000384[0] 1790100704.270891 mapwrite 0x4d7dd8 thru 0x4e0732 (0x895b bytes) 000385[0] 1790100704.349851 write 0x0f800e thru 0x103ef3 (0xbee6 bytes) 000386[0] 1790100704.361949 mapwrite 0x786314 thru 0x793dba (0xdaa7 bytes) 000388[0] 1790100704.524813 punch from 0x77d613 to 0x78087a (0x3268 bytes) 000389[1] 1790100704.564950 write 0x47ef96 thru 0x4827ea (0x3855 bytes) 000390[1] 1790100704.612974 punch from 0x32a12f to 0x337dc6 (0xdc98 bytes) 000391[1] 1790100704.673607 read 0x1bf399 thru 0x1ca7c5 (0xb42d bytes) 000394[1] 1790100704.717040 trunc from 0x793dbb to 0x6d5689 (0xbe731 bytes) 000398[0] 1790100704.781599 trunc from 0x6d568a to 0x16952f (0x56c15a bytes) 000399[0] 1790100704.851399 trunc from 0x169530 to 0x93019e (0x7c6c6f bytes) 000400[0] 1790100704.880389 read 0x779108 thru 0x782496 (0x938f bytes) 000401[0] 1790100704.920248 mapwrite 0x8592eb thru 0x85bd77 (0x2a8d bytes) 000402[1] 1790100705.014550 mapread 0x79eac9 thru 0x7aad98 (0xc2d0 bytes) 000406[1] 1790100705.125859 write 0x02d0db thru 0x0323c1 (0x52e7 bytes) 000407[0] 1790100705.157800 write 0x966867 thru 0x96d294 (0x6a2e bytes) 000408[1] 1790100705.181872 write 0x597176 thru 0x5a56bd (0xe548 bytes) 000409[0] 1790100705.224691 mapread 0x545ddb thru 0x553c4d (0xde73 bytes) 000411[0] 1790100705.323239 mapwrite 0x3d1d44 thru 0x3df603 (0xd8c0 bytes) 000412[0] 1790100705.520882 punch from 0x23f73d to 0x241911 (0x21d5 bytes) 000413[0] 1790100705.557570 write 0x57bc85 thru 0x57fc00 (0x3f7c bytes) 000418[1] 1790100705.663078 punch from 0x120e5c to 0x129434 (0x85d9 bytes) 000419[1] 1790100705.720510 read 0x829817 thru 0x833d6b (0xa555 bytes) 000420[0] 1790100705.795405 read 0x08c387 thru 0x08ee3b (0x2ab5 bytes) 000421[0] 1790100705.854994 trunc from 0x96d295 to 0x3d6c19 (0x59667b bytes) 000422[1] 1790100705.983153 punch from 0x2fa86d to 0x2fd355 (0x2ae9 bytes) 000426[1] 1790100706.084576 trunc from 0x3d6c1a to 0x99a7b7 (0x5c3b9e bytes) 000430[1] 1790100706.155517 read 0x0deabf thru 0x0ea0d8 (0xb61a bytes) 000431[1] 1790100706.167517 trunc from 0x99a7b8 to 0x784077 (0x216740 bytes) 000432[0] 1790100706.205429 write 0x98631b thru 0x994693 (0xe379 bytes) 000433[0] 1790100706.312971 punch from 0x89c086 to 0x8ab146 (0xf0c1 bytes) 000435[1] 1790100706.388799 mapread 0x68b069 thru 0x68d62a (0x25c2 bytes) 000436[1] 1790100706.460198 read 0x5dd451 thru 0x5df188 (0x1d38 bytes) 000437[1] 1790100706.493799 mapread 0x443146 thru 0x448dab (0x5c66 bytes) 000440[1] 1790100706.595866 read 0x3aa869 thru 0x3ac978 (0x2110 bytes) 000441[1] 1790100706.706835 trunc from 0x994694 to 0x8484fd (0x14c196 bytes) 000444[0] 1790100706.862235 mapwrite 0x33c759 thru 0x343df2 (0x769a bytes) 000445[1] 1790100707.114106 punch from 0x742722 to 0x7477d4 (0x50b3 bytes) 000446[0] 1790100707.143310 punch from 0x2b848c to 0x2bff54 (0x7ac9 bytes) 000448[0] 1790100707.189843 punch from 0x28d763 to 0x298dee (0xb68c bytes) 000449[0] 1790100707.213137 mapread 0x715df6 thru 0x724a53 (0xec5e bytes) 000451[1] 1790100707.296851 write 0x557229 thru 0x5657dd (0xe5b5 bytes) 000453[1] 1790100707.360894 read 0x6d6fed thru 0x6dc20a (0x521e bytes) 000454[0] 1790100707.388019 trunc from 0x8484fe to 0x7b4724 (0x93dd9 bytes) 000455[1] 1790100707.466135 trunc from 0x7b4725 to 0x2bb059 (0x4f96cb bytes) 000456[0] 1790100707.536919 punch from 0x0ecf29 to 0x0f8bd0 (0xbca8 bytes) 000458[1] 1790100707.596018 punch from 0x187ce6 to 0x18c56a (0x4885 bytes) 000461[0] 1790100707.664400 trunc from 0x2bb05a to 0x51c7d2 (0x261779 bytes) 000463[1] 1790100707.765570 mapread 0x389bf1 thru 0x38e9ff (0x4e0f bytes) 000466[1] 1790100707.848989 mapread 0x505b8b thru 0x509a34 (0x3eaa bytes) 000467[0] 1790100707.895881 write 0x10d689 thru 0x118172 (0xaaea bytes) 000468[1] 1790100707.943094 mapwrite 0x514c08 thru 0x524b9e (0xff97 bytes) 000469[1] 1790100708.075047 mapwrite 0x210cb7 thru 0x218a4f (0x7d99 bytes) 000471[0] 1790100708.169996 read 0x10dd73 thru 0x10e4a1 (0x72f bytes) 000472[0] 1790100708.204084 read 0x1614fb thru 0x170bcd (0xf6d3 bytes) 000474[1] 1790100708.218001 punch from 0x3f3bd5 to 0x3f42b0 (0x6dc bytes) 000476[0] 1790100708.237867 punch from 0x3454dc to 0x34cd72 (0x7897 bytes) 000477[1] 1790100708.270048 mapwrite 0x5ce508 thru 0x5dc1dd (0xdcd6 bytes) 000478[0] 1790100708.470175 read 0x37dcc8 thru 0x383cfc (0x6035 bytes) 000479[0] 1790100708.526015 mapwrite 0x9ee8f5 thru 0x9f56b1 (0x6dbd bytes) 000482[1] 1790100708.829695 trunc from 0x9f56b2 to 0x563a83 (0x491c2e bytes) 000483[1] 1790100708.947369 trunc from 0x563a84 to 0x13e114 (0x42596f bytes) 000486[1] 1790100709.113096 trunc from 0x13e115 to 0x313e68 (0x1d5d54 bytes) 000487[0] 1790100709.198924 punch from 0x111484 to 0x115099 (0x3c16 bytes) 000488[1] 1790100709.269904 trunc from 0x313e69 to 0x4332f7 (0x11f48f bytes) 000490[0] 1790100709.370906 read 0x2a68b1 thru 0x2ac961 (0x60b1 bytes) 000493[1] 1790100709.627477 mapread 0x37185d thru 0x37f730 (0xded4 bytes) 000494[0] 1790100709.756968 write 0x8db782 thru 0x8dd70e (0x1f8d bytes) 000495[0] 1790100709.835712 write 0x4eb7a7 thru 0x4fa04d (0xe8a7 bytes) 000496[0] 1790100709.913884 write 0x1feee1 thru 0x2055f0 (0x6710 bytes) 000497[1] 1790100709.988001 punch from 0x2fab5e to 0x3056fe (0xaba1 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 710 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x9a833e 000007[1] 1790100711.377678 trunc from 00000000 to 0x9a833d (0x9a833e bytes) 000012[1] 1790100711.468759 mapread 0x6aa000 thru 0x6ac051 (0x2052 bytes) 000013[0] 1790100711.501569 trunc from 0x9a833e to 0x557330 (0x45100d bytes) 000016[0] 1790100711.642394 write_OD 0x4cb000 thru 0x4d2fff (0x8000 bytes) 000018[1] 1790100711.760177 read 0x47d000 thru 0x486e3b (0x9e3c bytes) 000021[0] 1790100711.781634 read_OD 0x2da000 thru 0x2e7fff (0xe000 bytes) 000025[1] 1790100711.811198 mapread 0x31f000 thru 0x32c6f8 (0xd6f9 bytes) 000026[1] 1790100711.856603 mapwrite 0x8c3000 thru 0x8cbb06 (0x8b07 bytes) 000027[0] 1790100712.041093 read_OD 0x323000 thru 0x329fff (0x7000 bytes) 000028[0] 1790100712.091135 read_OD 0x81d000 thru 0x82bfff (0xf000 bytes) 000030[0] 1790100712.110325 mapread 0x638000 thru 0x6424f3 (0xa4f4 bytes) 000031[1] 1790100712.137725 trunc from 0x8cbb07 to 0x383422 (0x5486e4 bytes) 000032[0] 1790100712.200115 punch from 0x1b3eda to 0x1c017a (0xc2a1 bytes) 000034[0] 1790100712.248505 trunc from 0x383423 to 0x3cbf90 (0x48b6e bytes) 000035[0] 1790100712.281831 punch from 0x1d13fe to 0x1d4349 (0x2f4c bytes) 000037[0] 1790100712.330018 punch from 0x3a39b1 to 0x3aada8 (0x73f8 bytes) 000038[0] 1790100712.355907 write_OD 0x9e9000 thru 0x9f4fff (0xc000 bytes) 000039[1] 1790100712.478542 trunc from 0x9f5000 to 0x809efc (0x1eb103 bytes) 000041[1] 1790100712.615825 trunc from 0x809efd to 0x56dbcb (0x29c331 bytes) 000042[0] 1790100712.654248 trunc from 0x56dbcc to 0x03b97c (0x53224f bytes) 000043[0] 1790100712.713617 write_OD 0x360000 thru 0x366fff (0x7000 bytes) 000044[0] 1790100712.804129 write_OD 0x6a5000 thru 0x6abfff (0x7000 bytes) 000045[0] 1790100712.891953 write_OD 0x53d000 thru 0x540fff (0x4000 bytes) 000046[1] 1790100712.958836 mapwrite 0x69c000 thru 0x6a630d (0xa30e bytes) 000047[0] 1790100713.100981 write_OD 0x631000 thru 0x637fff (0x7000 bytes) 000048[1] 1790100713.189526 trunc from 0x6ac000 to 0x35dd20 (0x34e2df bytes) 000050[1] 1790100713.304232 read 0x142000 thru 0x1428d8 (0x8d9 bytes) 000054[1] 1790100713.467539 mapread 0x009000 thru 0x0165c1 (0xd5c2 bytes) 000055[0] 1790100713.468992 write_OD 0x80c000 thru 0x812fff (0x7000 bytes) 000056[1] 1790100713.584461 write 0x75e000 thru 0x76a455 (0xc456 bytes) 000058[1] 1790100713.732120 mapwrite 0x9fd000 thru 0x9ff28f (0x2290 bytes) 000059[1] 1790100713.905577 mapwrite 0x828000 thru 0x82f27a (0x727b bytes) 000060[1] 1790100714.015256 trunc from 0x9ff290 to 0x6a02bd (0x35efd2 bytes) 000062[0] 1790100714.080614 punch from 0x3b4b8d to 0x3b5c59 (0x10cd bytes) 000063[1] 1790100714.112552 mapwrite 0x430000 thru 0x43c4f8 (0xc4f9 bytes) 000064[0] 1790100714.340964 mapread 0x451000 thru 0x45420f (0x3210 bytes) 000065[0] 1790100714.388580 write_OD 0x367000 thru 0x36cfff (0x6000 bytes) 000067[0] 1790100714.463327 write_OD 0x4b9000 thru 0x4c2fff (0xa000 bytes) 000068[0] 1790100714.537989 mapread 0x444000 thru 0x445139 (0x113a bytes) 000069[1] 1790100714.558609 mapwrite 0x029000 thru 0x02abeb (0x1bec bytes) 000071[1] 1790100714.648102 trunc from 0x6a02be to 0x4efe4b (0x1b0472 bytes) 000073[0] 1790100714.734205 punch from 0x32b271 to 0x32f37e (0x410e bytes) 000075[1] 1790100714.796706 mapwrite 0x0ad000 thru 0x0b49db (0x79dc bytes) 000076[0] 1790100714.959386 read_OD 0x114000 thru 0x121fff (0xe000 bytes) 000077[1] 1790100714.999227 mapwrite 0x333000 thru 0x3422d5 (0xf2d6 bytes) 000079[0] 1790100715.335714 trunc from 0x4efe4c to 0x11d216 (0x3d2c35 bytes) 000082[1] 1790100715.387637 mapread 0x071000 thru 0x0727fc (0x17fd bytes) 000083[0] 1790100715.426735 mapwrite 0x5ea000 thru 0x5f246e (0x846f bytes) 000088[0] 1790100715.788720 mapwrite 0x4dc000 thru 0x4e7f7a (0xbf7b bytes) 000089[0] 1790100715.905204 punch from 0x040b2d to 0x04fe87 (0xf35b bytes) 000090[1] 1790100715.909673 trunc from 0x5f246f to 0x3c75a7 (0x22aec7 bytes) 000092[1] 1790100715.991368 trunc from 0x3c75a8 to 0x6c750f (0x2fff68 bytes) 000094[1] 1790100716.070570 mapread 0x56b000 thru 0x575207 (0xa208 bytes) 000098[1] 1790100716.226638 mapwrite 0x129000 thru 0x130c42 (0x7c43 bytes) 000100[0] 1790100716.461704 mapwrite 0x369000 thru 0x373e39 (0xae3a bytes) 000101[0] 1790100716.602911 mapread 0x0b2000 thru 0x0b6951 (0x4952 bytes) 000103[0] 1790100716.648456 write_OD 0x42c000 thru 0x437fff (0xc000 bytes) 000104[1] 1790100716.730724 read 0x2d6000 thru 0x2da89c (0x489d bytes) 000106[1] 1790100716.745896 write 0x150000 thru 0x150786 (0x787 bytes) 000110[0] 1790100716.822411 trunc from 0x6c7510 to 0x8320ac (0x16ab9d bytes) 000114[1] 1790100716.910594 read 0x1f5000 thru 0x200d3b (0xbd3c bytes) 000115[1] 1790100716.953848 trunc from 0x8320ad to 0x6249e0 (0x20d6cc bytes) 000116[1] 1790100717.024834 write 0x2b1000 thru 0x2bd4e6 (0xc4e7 bytes) 000117[1] 1790100717.052207 mapread 0x60c000 thru 0x613405 (0x7406 bytes) 000118[1] 1790100717.093173 punch from 0x1950c8 to 0x19ce4f (0x7d88 bytes) 000119[0] 1790100717.122171 mapwrite 0x516000 thru 0x5175b7 (0x15b8 bytes) 000120[1] 1790100717.244648 read 0x1b5000 thru 0x1b69f1 (0x19f2 bytes) 000122[0] 1790100717.281168 mapwrite 0x324000 thru 0x32ef9b (0xaf9c bytes) 000123[1] 1790100717.411293 punch from 0x1f5069 to 0x1fbc68 (0x6c00 bytes) 000124[0] 1790100717.429212 trunc from 0x6249e1 to 0x57be2d (0xa8bb3 bytes) 000125[0] 1790100717.519095 punch from 0x04bc43 to 0x04dcd7 (0x2095 bytes) 000126[0] 1790100717.541190 mapwrite 0x963000 thru 0x96f38e (0xc38f bytes) 000127[1] 1790100717.691759 trunc from 0x96f38f to 0x943b92 (0x2b7fc bytes) 000128[1] 1790100717.733092 mapread 0x057000 thru 0x0634cb (0xc4cc bytes) 000129[0] 1790100717.783369 mapread 0x6fa000 thru 0x6fdc27 (0x3c28 bytes) 000130[1] 1790100717.852784 trunc from 0x943b93 to 0x2f96b4 (0x64a4de bytes) 000131[0] 1790100717.939152 trunc from 0x2f96b5 to 0x7b1009 (0x4b7955 bytes) 000134[0] 1790100718.034043 read_OD 0x33c000 thru 0x345fff (0xa000 bytes) 000135[0] 1790100718.063831 write_OD 0x713000 thru 0x713fff (0x1000 bytes) 000136[1] 1790100718.131030 read 0x1ff000 thru 0x20227f (0x3280 bytes) 000137[1] 1790100718.137726 read 0x54b000 thru 0x555bda (0xabdb bytes) 000139[0] 1790100718.152159 trunc from 0x7b100a to 0x86717d (0xb6174 bytes) 000140[0] 1790100718.213805 mapread 0x07c000 thru 0x08766a (0xb66b bytes) 000141[1] 1790100718.276880 trunc from 0x86717e to 0x62d9a4 (0x2397d9 bytes) 000142[0] 1790100718.347438 punch from 0x09987e to 0x09f342 (0x5ac5 bytes) 000143[0] 1790100718.368625 mapwrite 0x97a000 thru 0x984c48 (0xac49 bytes) 000144[1] 1790100718.562068 trunc from 0x984c49 to 0x095871 (0x8ef3d7 bytes) 000145[1] 1790100718.601116 trunc from 0x095872 to 0x090213 (0x565e bytes) 000146[0] 1790100718.632962 trunc from 0x090214 to 0x14211e (0xb1f0b bytes) 000147[0] 1790100718.694354 mapwrite 0x6fe000 thru 0x709d10 (0xbd11 bytes) 000148[1] 1790100718.830111 write 0x5d0000 thru 0x5de73d (0xe73e bytes) 000149[1] 1790100718.857414 mapread 0x42a000 thru 0x4389d4 (0xe9d5 bytes) 000150[1] 1790100718.957444 punch from 0x4fe3b3 to 0x50249f (0x40ed bytes) truncating to largest ever: 0x9fd9cb 000152[0] 1790100718.996496 trunc from 0x709d11 to 0x9fd9ca (0x2f3cba bytes) 000153[0] 1790100719.049899 read_OD 0x1b3000 thru 0x1bbfff (0x9000 bytes) 000155[1] 1790100719.077177 trunc from 0x9fd9cb to 0x64b366 (0x3b2664 bytes) 000157[1] 1790100719.116544 read 0x2e0000 thru 0x2e4eb7 (0x4eb8 bytes) 000161[1] 1790100719.176911 read 0x446000 thru 0x44c6f7 (0x66f8 bytes) 000164[0] 1790100719.238257 trunc from 0x64b367 to 0x71776f (0xcc409 bytes) 000167[1] 1790100719.356724 write 0x2ca000 thru 0x2d34b2 (0x94b3 bytes) 000169[1] 1790100719.433045 mapread 0x3de000 thru 0x3e32d0 (0x52d1 bytes) 000171[0] 1790100719.486923 write_OD 0x81d000 thru 0x829fff (0xd000 bytes) 000172[1] 1790100719.646796 mapread 0x16b000 thru 0x1762f8 (0xb2f9 bytes) 000175[0] 1790100719.734240 mapwrite 0x49b000 thru 0x4aa793 (0xf794 bytes) 000176[0] 1790100719.943664 mapread 0x242000 thru 0x24f8e5 (0xd8e6 bytes) 000177[0] 1790100720.018933 read_OD 0x5c6000 thru 0x5cafff (0x5000 bytes) 000178[1] 1790100720.079869 trunc from 0x82a000 to 0x70875b (0x1218a4 bytes) 000181[0] 1790100720.203919 read_OD 0x26d000 thru 0x275fff (0x9000 bytes) 000184[1] 1790100720.265426 write 0x238000 thru 0x24785c (0xf85d bytes) 000185[1] 1790100720.309479 write 0x64d000 thru 0x655dcd (0x8dce bytes) 000186[0] 1790100720.323572 write_OD 0x98a000 thru 0x997fff (0xe000 bytes) 000187[0] 1790100720.418945 write_OD 0x581000 thru 0x583fff (0x3000 bytes) 000191[1] 1790100720.479662 read 0x72e000 thru 0x73a6d0 (0xc6d1 bytes) 000192[1] 1790100720.489881 mapwrite 0x5df000 thru 0x5e3811 (0x4812 bytes) 000193[0] 1790100720.593448 mapwrite 0x12f000 thru 0x1340d1 (0x50d2 bytes) 000195[1] 1790100720.785936 mapwrite 0x0aa000 thru 0x0b8271 (0xe272 bytes) 000196[1] 1790100720.967695 read 0x88a000 thru 0x88c54c (0x254d bytes) 000197[1] 1790100721.007165 mapwrite 0x006000 thru 0x00d4a0 (0x74a1 bytes) 000201[1] 1790100721.288408 trunc from 0x998000 to 0x189585 (0x80ea7a bytes) 000202[0] 1790100721.316187 mapread 0x083000 thru 0x088082 (0x5083 bytes) 000203[1] 1790100721.376900 write 0x0e3000 thru 0x0efe4d (0xce4e bytes) 000204[1] 1790100721.404688 punch from 0x04ad82 to 0x04c225 (0x14a4 bytes) 000209[1] 1790100721.490605 mapread 0x034000 thru 0x03f5b4 (0xb5b5 bytes) 000211[1] 1790100721.552671 mapwrite 0x5a0000 thru 0x5a8125 (0x8126 bytes) 000213[0] 1790100721.774006 punch from 0x1ebd35 to 0x1fb6bc (0xf988 bytes) 000214[0] 1790100721.830710 trunc from 0x5a8126 to 0x542d14 (0x65411 bytes) 000215[1] 1790100721.857671 trunc from 0x542d15 to 0x7b601f (0x27330b bytes) 000216[0] 1790100721.915636 mapwrite 0x09c000 thru 0x09ee8b (0x2e8c bytes) 000217[1] 1790100722.036086 trunc from 0x7b6020 to 0x257499 (0x55eb86 bytes) 000218[0] 1790100722.129463 read_OD 0x211000 thru 0x21efff (0xe000 bytes) 000219[0] 1790100722.268202 mapwrite 0x556000 thru 0x56139f (0xb3a0 bytes) 000221[1] 1790100722.629315 write 0x397000 thru 0x3974f5 (0x4f6 bytes) 000224[1] 1790100722.705320 mapread 0x326000 thru 0x330226 (0xa227 bytes) 000227[0] 1790100722.781565 write_OD 0x946000 thru 0x951fff (0xc000 bytes) 000229[1] 1790100722.862828 trunc from 0x952000 to 0x950831 (0x17ce bytes) 000230[0] 1790100722.917411 punch from 0x8d081f to 0x8d2c77 (0x2459 bytes) 000231[0] 1790100722.947773 read_OD 0x576000 thru 0x576fff (0x1000 bytes) 000232[0] 1790100722.971815 punch from 0x55de12 to 0x5697cc (0xb9bb bytes) 000233[1] 1790100723.040497 punch from 0x741d04 to 0x74e679 (0xc976 bytes) 000234[0] 1790100723.100804 trunc from 0x950832 to 0x2abcdd (0x6a4b54 bytes) 000236[0] 1790100723.201243 mapread 0x0a1000 thru 0x0a6738 (0x5739 bytes) 000240[0] 1790100723.253431 punch from 0x1ed127 to 0x1ee6b4 (0x158e bytes) 000243[1] 1790100723.331704 punch from 0x150767 to 0x153f30 (0x37ca bytes) 000247[1] 1790100723.546169 punch from 0x0b0ff8 to 0x0b3f7d (0x2f86 bytes) 000249[0] 1790100723.615620 read_OD 0x051000 thru 0x051fff (0x1000 bytes) 000250[1] 1790100723.645936 read 0x167000 thru 0x171e22 (0xae23 bytes) 000251[1] 1790100723.699026 mapread 0x262000 thru 0x26ca98 (0xaa99 bytes) 000256[1] 1790100723.704805 trunc from 0x2abcde to 0x486f34 (0x1db257 bytes) 000263[1] 1790100723.792031 write 0x082000 thru 0x08d8a3 (0xb8a4 bytes) 000265[1] 1790100723.869491 trunc from 0x486f35 to 0x44a0f6 (0x3ce3e bytes) 000268[0] 1790100723.926620 trunc from 0x44a0f7 to 0x68a9c8 (0x2408d2 bytes) 000269[0] 1790100724.026175 mapread 0x526000 thru 0x53239d (0xc39e bytes) 000270[1] 1790100724.226411 mapwrite 0x6b5000 thru 0x6bfd09 (0xad0a bytes) 000273[1] 1790100724.665792 mapwrite 0x6d4000 thru 0x6d5bbf (0x1bc0 bytes) 000274[1] 1790100724.822107 read 0x0c1000 thru 0x0cd035 (0xc036 bytes) 000275[0] 1790100724.834758 trunc from 0x6d5bc0 to 0x38872d (0x34d492 bytes) 000276[1] 1790100724.911952 mapread 0x287000 thru 0x28970d (0x270e bytes) 000277[0] 1790100725.010818 punch from 0x123ff1 to 0x1264d8 (0x24e8 bytes) 000278[1] 1790100725.032682 read 0x21e000 thru 0x227ed4 (0x9ed5 bytes) 000282[0] 1790100725.155526 write_OD 0x232000 thru 0x23ffff (0xe000 bytes) 000283[1] 1790100725.238783 mapread 0x1cb000 thru 0x1d24e9 (0x74ea bytes) 000285[0] 1790100725.344248 trunc from 0x38872e to 0x29a781 (0xedfac bytes) 000287[0] 1790100725.417919 trunc from 0x29a782 to 0x61ab80 (0x3803ff bytes) 000288[1] 1790100725.461933 mapread 0x3d3000 thru 0x3d3146 (0x147 bytes) 000293[0] 1790100725.630313 punch from 0x15055c to 0x15efd6 (0xea7b bytes) 000294[0] 1790100725.707677 mapread 0x247000 thru 0x252dbe (0xbdbf bytes) 000296[1] 1790100725.834426 write 0x9ef000 thru 0x9f6779 (0x777a bytes) 000297[1] 1790100725.858869 mapread 0x9df000 thru 0x9e095f (0x1960 bytes) 000298[1] 1790100725.916903 trunc from 0x9f677a to 0x13c29a (0x8ba4df bytes) 000299[0] 1790100725.984579 read_OD 0x096000 thru 0x098fff (0x3000 bytes) 000301[0] 1790100726.029258 punch from 0x009999 to 0x015020 (0xb688 bytes) 000303[0] 1790100726.058248 mapwrite 0x782000 thru 0x78c72f (0xa730 bytes) 000304[1] 1790100726.516062 read 0x258000 thru 0x266c08 (0xec09 bytes) 000305[1] 1790100726.578178 write 0x433000 thru 0x43a989 (0x798a bytes) 000307[1] 1790100726.622150 write 0x4e1000 thru 0x4e4308 (0x3309 bytes) 000309[1] 1790100726.635006 mapread 0x15f000 thru 0x1606b0 (0x16b1 bytes) 000316[0] 1790100726.765790 mapwrite 0x06e000 thru 0x072548 (0x4549 bytes) 000319[0] 1790100727.026133 read_OD 0x43f000 thru 0x444fff (0x6000 bytes) 000320[1] 1790100727.050840 read 0x39a000 thru 0x3a45ff (0xa600 bytes) 000321[1] 1790100727.067199 read 0x215000 thru 0x220bcf (0xbbd0 bytes) 000322[0] 1790100727.075967 punch from 0x32efcd to 0x33de81 (0xeeb5 bytes) 000325[0] 1790100727.172519 punch from 0x2c4cbb to 0x2ca64d (0x5993 bytes) 000326[1] 1790100727.194989 mapwrite 0x5b8000 thru 0x5c1674 (0x9675 bytes) 000329[0] 1790100727.330198 punch from 0x427b16 to 0x42ced0 (0x53bb bytes) 000330[0] 1790100727.364219 trunc from 0x78c730 to 0x7bf8df (0x331b0 bytes) 000332[0] 1790100727.425005 trunc from 0x7bf8e0 to 0x97739b (0x1b7abc bytes) 000335[1] 1790100727.472332 write 0x869000 thru 0x8727af (0x97b0 bytes) 000337[0] 1790100727.573750 trunc from 0x97739c to 0x2c44f1 (0x6b2eaa bytes) 000338[0] 1790100727.666228 trunc from 0x2c44f2 to 0x3f3044 (0x12eb53 bytes) 000339[0] 1790100727.737245 read_OD 0x1c4000 thru 0x1cbfff (0x8000 bytes) 000342[1] 1790100727.800740 punch from 0x0e2147 to 0x0e8e26 (0x6ce0 bytes) 000343[0] 1790100727.829466 mapwrite 0x19e000 thru 0x1aa7c6 (0xc7c7 bytes) 000344[1] 1790100728.062764 read 0x398000 thru 0x3a6dfa (0xedfb bytes) 000346[0] 1790100728.197124 read_OD 0x280000 thru 0x285fff (0x6000 bytes) 000347[0] 1790100728.205350 write_OD 0x50a000 thru 0x50dfff (0x4000 bytes) 000348[1] 1790100728.272825 mapread 0x347000 thru 0x35253c (0xb53d bytes) 000349[0] 1790100728.346659 mapread 0x437000 thru 0x43ba00 (0x4a01 bytes) 000350[0] 1790100728.466472 mapwrite 0x9e9000 thru 0x9f19a5 (0x89a6 bytes) 000352[0] 1790100728.685241 mapread 0x93a000 thru 0x93a483 (0x484 bytes) 000353[1] 1790100728.706595 mapwrite 0x153000 thru 0x157478 (0x4479 bytes) 000354[0] 1790100728.804595 mapread 0x8fc000 thru 0x9064f0 (0xa4f1 bytes) 000359[0] 1790100728.885781 read_OD 0x96d000 thru 0x974fff (0x8000 bytes) 000360[0] 1790100728.896613 punch from 0x261734 to 0x26d9df (0xc2ac bytes) 000361[1] 1790100728.931021 mapwrite 0x85d000 thru 0x862c5c (0x5c5d bytes) 000362[1] 1790100729.238598 read 0x911000 thru 0x914a0e (0x3a0f bytes) 000368[0] 1790100729.278182 mapwrite 0x82a000 thru 0x82c5d1 (0x25d2 bytes) 000370[0] 1790100729.500824 mapread 0x082000 thru 0x0874a3 (0x54a4 bytes) 000372[1] 1790100729.556557 punch from 0x5498bb to 0x556c0c (0xd352 bytes) 000376[0] 1790100729.652192 write_OD 0x085000 thru 0x093fff (0xf000 bytes) 000377[1] 1790100729.730761 trunc from 0x9f19a6 to 0x448fb5 (0x5a89f0 bytes) 000380[1] 1790100729.892521 write 0x2d8000 thru 0x2d934b (0x134c bytes) 000382[1] 1790100729.943893 trunc from 0x448fb6 to 0x604b0f (0x1bbb5a bytes) 000383[0] 1790100729.987052 read_OD 0x18f000 thru 0x193fff (0x5000 bytes) 000384[1] 1790100730.038895 write 0x980000 thru 0x986113 (0x6114 bytes) 000386[1] 1790100730.117680 write 0x102000 thru 0x111661 (0xf662 bytes) 000387[0] 1790100730.139750 read_OD 0x765000 thru 0x76dfff (0x9000 bytes) 000388[1] 1790100730.178091 trunc from 0x986114 to 0x0fab8a (0x88b589 bytes) 000389[1] 1790100730.211945 trunc from 0x0fab8b to 0x647786 (0x54cbfc bytes) 000390[0] 1790100730.230943 read_OD 0x55e000 thru 0x561fff (0x4000 bytes) 000391[0] 1790100730.256034 read_OD 0x440000 thru 0x442fff (0x3000 bytes) 000392[0] 1790100730.275080 trunc from 0x647787 to 0x142c5e (0x504b28 bytes) 000396[0] 1790100730.365539 mapread 0x073000 thru 0x07d5e0 (0xa5e1 bytes) 000397[1] 1790100730.455883 mapwrite 0x0e5000 thru 0x0eb236 (0x6237 bytes) 000398[1] 1790100730.636344 trunc from 0x142c5f to 0x6066b9 (0x4c3a5b bytes) 000399[0] 1790100730.688954 trunc from 0x6066ba to 0x58a8f5 (0x7bdc4 bytes) 000400[0] 1790100730.731118 write_OD 0x1bf000 thru 0x1c8fff (0xa000 bytes) 000401[1] 1790100730.801501 mapread 0x074000 thru 0x07cac7 (0x8ac8 bytes) 000404[0] 1790100730.816431 trunc from 0x58a8f6 to 0x7f8244 (0x26d94f bytes) 000405[0] 1790100730.866817 mapwrite 0x633000 thru 0x635a7d (0x2a7e bytes) 000411[0] 1790100730.982799 mapread 0x684000 thru 0x691076 (0xd077 bytes) 000412[1] 1790100731.033958 trunc from 0x7f8245 to 0x102966 (0x6f58de bytes) 000413[1] 1790100731.074243 mapwrite 0x121000 thru 0x12f01c (0xe01d bytes) 000414[0] 1790100731.179237 write_OD 0x0fc000 thru 0x104fff (0x9000 bytes) 000417[0] 1790100731.271367 mapwrite 0x7a5000 thru 0x7aad14 (0x5d15 bytes) 000418[1] 1790100731.416260 mapwrite 0x5b3000 thru 0x5bae6c (0x7e6d bytes) 000422[0] 1790100731.676758 punch from 0x64698e to 0x655f28 (0xf59b bytes) 000424[1] 1790100731.740581 read 0x189000 thru 0x189052 (0x53 bytes) 000425[1] 1790100731.770136 mapwrite 0x4b4000 thru 0x4bca98 (0x8a99 bytes) 000426[1] 1790100731.866588 mapwrite 0x2ee000 thru 0x2fbfb3 (0xdfb4 bytes) 000430[1] 1790100732.058019 punch from 0x3bd8b2 to 0x3c5b41 (0x8290 bytes) 000431[0] 1790100732.068588 punch from 0x59466e to 0x594a41 (0x3d4 bytes) 000432[0] 1790100732.088145 mapwrite 0x92e000 thru 0x932f1f (0x4f20 bytes) 000433[0] 1790100732.232734 punch from 0x5ee3be to 0x5f6df6 (0x8a39 bytes) 000435[0] 1790100732.285017 mapread 0x1b0000 thru 0x1b2557 (0x2558 bytes) 000437[1] 1790100732.369617 punch from 0x2c71bb to 0x2c88e4 (0x172a bytes) 000438[1] 1790100732.401207 read 0x047000 thru 0x04e6b9 (0x76ba bytes) 000441[0] 1790100732.464631 punch from 0x2fdb82 to 0x308564 (0xa9e3 bytes) 000442[1] 1790100732.503731 punch from 0x34ddfd to 0x350d76 (0x2f7a bytes) 000443[0] 1790100732.523763 mapread 0x20d000 thru 0x20fd5c (0x2d5d bytes) 000444[0] 1790100732.561450 punch from 0x69e58a to 0x6a72ab (0x8d22 bytes) 000445[0] 1790100732.581739 read_OD 0x055000 thru 0x05afff (0x6000 bytes) 000447[1] 1790100732.605582 mapwrite 0x666000 thru 0x674984 (0xe985 bytes) 000449[1] 1790100732.913239 read 0x3ca000 thru 0x3cd9f9 (0x39fa bytes) 000450[0] 1790100732.982583 mapwrite 0x1e3000 thru 0x1eb54e (0x854f bytes) 000451[1] 1790100733.234152 read 0x6b0000 thru 0x6b726f (0x7270 bytes) 000454[0] 1790100733.294328 punch from 0x1af669 to 0x1b7ba3 (0x853b bytes) 000455[1] 1790100733.330016 trunc from 0x932f20 to 0x9abbb6 (0x78c97 bytes) 000456[0] 1790100733.401121 write_OD 0x3bb000 thru 0x3c7fff (0xd000 bytes) 000458[1] 1790100733.487657 write 0x0fa000 thru 0x104861 (0xa862 bytes) 000459[0] 1790100733.521661 trunc from 0x9abbb7 to 0x4d62f1 (0x4d58c5 bytes) 000460[1] 1790100733.596654 write 0x81f000 thru 0x82cc55 (0xdc56 bytes) 000462[0] 1790100733.633458 trunc from 0x82cc56 to 0x533f1e (0x2f8d37 bytes) 000465[0] 1790100733.747963 trunc from 0x533f1f to 0x36d3ea (0x1c6b34 bytes) 000467[1] 1790100733.800755 write 0x232000 thru 0x23c224 (0xa225 bytes) 000468[1] 1790100733.846731 mapwrite 0x84f000 thru 0x858b32 (0x9b33 bytes) 000469[0] 1790100734.099514 read_OD 0x646000 thru 0x64bfff (0x6000 bytes) 000470[1] 1790100734.144432 punch from 0x541b8c to 0x547089 (0x54fe bytes) 000472[1] 1790100734.225077 read 0x163000 thru 0x167aa4 (0x4aa5 bytes) 000473[1] 1790100734.240344 mapwrite 0x21d000 thru 0x222865 (0x5866 bytes) 000474[1] 1790100734.394575 punch from 0x3d36dc to 0x3da61a (0x6f3f bytes) 000475[1] 1790100734.426190 punch from 0x6b9046 to 0x6c838e (0xf349 bytes) 000476[1] 1790100734.434963 read 0x7df000 thru 0x7e08be (0x18bf bytes) 000477[0] 1790100734.458199 write_OD 0x728000 thru 0x72afff (0x3000 bytes) 000478[1] 1790100734.530586 punch from 0x03bc15 to 0x04ac1e (0xf00a bytes) 000482[0] 1790100734.596506 punch from 0x4885b4 to 0x494d9d (0xc7ea bytes) 000484[0] 1790100734.631575 write_OD 0x09d000 thru 0x0a7fff (0xb000 bytes) 000485[1] 1790100734.703990 mapread 0x2cd000 thru 0x2d479a (0x779b bytes) 000486[1] 1790100734.738579 mapwrite 0x66b000 thru 0x6757e0 (0xa7e1 bytes) 000487[1] 1790100734.870597 read 0x581000 thru 0x587953 (0x6954 bytes) 000490[1] 1790100734.897498 trunc from 0x858b33 to 0x2ae155 (0x5aa9dd bytes) 000491[0] 1790100734.924024 trunc from 0x2ae156 to 0x09249f (0x21bcb6 bytes) 000492[1] 1790100734.969161 trunc from 0x0924a0 to 0x4b6d30 (0x424891 bytes) 000493[1] 1790100734.999226 mapread 0x061000 thru 0x070e47 (0xfe48 bytes) 000494[1] 1790100735.104687 write 0x93d000 thru 0x944518 (0x7519 bytes) 000495[0] 1790100735.113134 mapwrite 0x028000 thru 0x0364d0 (0xe4d1 bytes) 000498[0] 1790100735.345416 trunc from 0x944519 to 0x96a087 (0x25b6f bytes) 000500[1] 1790100735.387850 punch from 0x2744bd to 0x275a65 (0x15a9 bytes) All operations completed A-OK! PASS 16a (62s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:12:21 (1790100741) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.423201 s, 19.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.296762 s, 28.3 MB/s Chance of close/open is 1 in 50 Seed set to 744 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1ad6da 000003[1] 1790100745.183542 trunc from 00000000 to 0x1ad6d9 (0x1ad6da bytes) 000004[1] 1790100745.237900 write 0x5f2a88 thru 0x5f40be (0x1637 bytes) 000006[0] 1790100745.276917 punch from 0x38e49f to 0x38e778 (0x2da bytes) 000007[0] 1790100745.316626 write 0x8627c4 thru 0x863bda (0x1417 bytes) 000008[1] 1790100745.339815 mapwrite 0x5edbc6 thru 0x5eddcb (0x206 bytes) 000009[0] 1790100745.447741 write 0x047e0a thru 0x049325 (0x151c bytes) 000010[0] 1790100745.475106 mapwrite 0x90f492 thru 0x91010c (0xc7b bytes) 000014[1] 1790100745.719908 write 0x6faf6b thru 0x6fb4be (0x554 bytes) 000015[0] 1790100745.771403 mapwrite 0x7de835 thru 0x7df39c (0xb68 bytes) 000016[0] 1790100745.853729 mapwrite 0x092083 thru 0x093583 (0x1501 bytes) 000017[0] 1790100745.938060 punch from 0x3699f5 to 0x36b66c (0x1c78 bytes) 000020[0] 1790100746.005464 write 0x2f3ed1 thru 0x2f472c (0x85c bytes) 000021[1] 1790100746.015617 read 0x895bdc thru 0x8978be (0x1ce3 bytes) 000022[0] 1790100746.052430 mapread 0x9069e9 thru 0x90745a (0xa72 bytes) 000024[0] 1790100746.095194 read 0x688319 thru 0x689d7a (0x1a62 bytes) 000025[0] 1790100746.120082 punch from 0x8d9330 to 0x8da1b2 (0xe83 bytes) 000026[0] 1790100746.229833 read 0x2dcd7a thru 0x2de1e6 (0x146d bytes) 000028[1] 1790100746.236409 mapread 0x0630c0 thru 0x064d6a (0x1cab bytes) 000031[0] 1790100746.340384 write 0x20b447 thru 0x20cfaa (0x1b64 bytes) 000032[1] 1790100746.405805 read 0x1bc3b1 thru 0x1bc7de (0x42e bytes) 000033[0] 1790100746.520730 mapwrite 0x7b6834 thru 0x7b6885 (0x52 bytes) 000035[1] 1790100746.688905 punch from 0x4e109f to 0x4e29a1 (0x1903 bytes) 000036[1] 1790100746.714854 read 0x28ed76 thru 0x28f2df (0x56a bytes) 000037[0] 1790100746.742185 trunc from 0x91010d to 0x0120c6 (0x8fe046 bytes) 000038[0] 1790100746.805937 write 0x94d191 thru 0x94e1de (0x104e bytes) 000040[0] 1790100746.843936 mapwrite 0x025db7 thru 0x02792e (0x1b78 bytes) 000041[1] 1790100746.901163 mapwrite 0x00460f thru 0x006580 (0x1f72 bytes) 000042[1] 1790100747.009809 read 0x203f0c thru 0x20546d (0x1562 bytes) 000043[0] 1790100747.037908 write 0x451e13 thru 0x4527e5 (0x9d3 bytes) truncating to largest ever: 0x9d39d7 000046[0] 1790100747.103226 trunc from 0x94e1df to 0x9d39d6 (0x857f8 bytes) 000047[1] 1790100747.167587 mapwrite 0x9acb5c thru 0x9ad4e6 (0x98b bytes) 000048[1] 1790100747.275460 write 0x14582e thru 0x14619a (0x96d bytes) 000051[1] 1790100747.336651 mapread 0x819a6d thru 0x81a58a (0xb1e bytes) 000052[0] 1790100747.376615 mapwrite 0x94a7a8 thru 0x94c28a (0x1ae3 bytes) truncating to largest ever: 0x9d4c3a 000053[0] 1790100747.499665 trunc from 0x9d39d7 to 0x9d4c39 (0x1263 bytes) 000054[0] 1790100747.554384 read 0x7c867d thru 0x7ca46b (0x1def bytes) 000055[1] 1790100747.579694 mapwrite 0x064d22 thru 0x066b5b (0x1e3a bytes) 000060[1] 1790100747.765977 write 0x9e9515 thru 0x9ea691 (0x117d bytes) 000061[0] 1790100747.783665 trunc from 0x9ea692 to 0x1a34f0 (0x8471a1 bytes) 000063[0] 1790100747.827224 trunc from 0x1a34f1 to 0x5c1258 (0x41dd68 bytes) 000064[1] 1790100747.855429 trunc from 0x5c1259 to 0x78bcb5 (0x1caa5d bytes) 000067[0] 1790100747.903047 read 0x142a2d thru 0x14312f (0x703 bytes) 000068[1] 1790100747.956527 trunc from 0x78bcb6 to 0x51195f (0x27a356 bytes) 000069[0] 1790100748.046709 punch from 0x00e79c to 0x00f248 (0xaad bytes) 000071[1] 1790100748.137811 punch from 0x17ae81 to 0x17c193 (0x1313 bytes) 000072[1] 1790100748.171413 trunc from 0x511960 to 0x20b239 (0x306726 bytes) 000074[1] 1790100748.234968 punch from 0x058b9f to 0x058df5 (0x257 bytes) 000075[1] 1790100748.280111 mapread 0x1ab1e2 thru 0x1ac058 (0xe77 bytes) 000076[1] 1790100748.325767 punch from 0x1a835d to 0x1a8cda (0x97e bytes) 000077[0] 1790100748.342443 mapwrite 0x99379a thru 0x994271 (0xad8 bytes) 000081[0] 1790100748.601478 read 0x2c4096 thru 0x2c58d3 (0x183e bytes) 000082[1] 1790100748.665529 mapread 0x304e42 thru 0x305fa4 (0x1163 bytes) 000083[1] 1790100748.741977 read 0x1eeed4 thru 0x1efcce (0xdfb bytes) 000084[0] 1790100748.768071 mapread 0x79e3f0 thru 0x79f6d4 (0x12e5 bytes) 000085[0] 1790100748.800965 trunc from 0x994272 to 0x201a23 (0x79284e bytes) 000086[1] 1790100748.844144 read 0x03f28b thru 0x0402e3 (0x1059 bytes) 000088[0] 1790100748.912970 trunc from 0x201a24 to 0x5315ce (0x32fbab bytes) 000089[0] 1790100748.957166 trunc from 0x5315cf to 0x25b55e (0x2d6070 bytes) 000094[1] 1790100749.082175 punch from 0x0bbf47 to 0x0bde11 (0x1ecb bytes) 000096[1] 1790100749.145132 read 0x003c14 thru 0x005c10 (0x1ffd bytes) 000099[0] 1790100749.273558 trunc from 0x25b55f to 0x9d1c09 (0x7766ab bytes) 000101[0] 1790100749.317620 read 0x3461dc thru 0x346fbe (0xde3 bytes) 000103[0] 1790100749.325007 read 0x36952d thru 0x36a187 (0xc5b bytes) 000106[0] 1790100749.359130 write 0x973b0a thru 0x975288 (0x177f bytes) 000107[0] 1790100749.446407 trunc from 0x9d1c0a to 0x8d568c (0xfc57d bytes) 000108[0] 1790100749.572485 mapread 0x30809a thru 0x308fc7 (0xf2e bytes) 000109[1] 1790100749.618879 write 0x053736 thru 0x054eac (0x1777 bytes) 000110[1] 1790100749.751304 read 0x464f70 thru 0x465799 (0x82a bytes) 000111[1] 1790100749.772450 read 0x68ad01 thru 0x68cb95 (0x1e95 bytes) 000112[0] 1790100749.778419 mapwrite 0x0a0a0b thru 0x0a1e4c (0x1442 bytes) 000113[0] 1790100749.878798 mapread 0x17e5a9 thru 0x17f5ab (0x1003 bytes) 000115[1] 1790100749.907470 mapread 0x34c081 thru 0x34ca87 (0xa07 bytes) 000120[0] 1790100749.945488 read 0x7f63f9 thru 0x7f7396 (0xf9e bytes) 000121[1] 1790100749.951808 punch from 0x14cb50 to 0x14e827 (0x1cd8 bytes) 000122[0] 1790100749.979617 write 0x1f8edf thru 0x1fa7c4 (0x18e6 bytes) 000123[0] 1790100750.024680 trunc from 0x8d568d to 0x121d0b (0x7b3981 bytes) 000124[0] 1790100750.067896 mapread 0x0cbe71 thru 0x0cce25 (0xfb5 bytes) 000125[1] 1790100750.104552 mapread 0x064093 thru 0x0645be (0x52c bytes) 000126[1] 1790100750.185807 write 0x727cc9 thru 0x728000 (0x338 bytes) 000127[0] 1790100750.203524 mapwrite 0x0a298e thru 0x0a48c4 (0x1f37 bytes) 000129[0] 1790100750.457529 read 0x177172 thru 0x17810f (0xf9e bytes) 000130[0] 1790100750.506936 read 0x3ba0b3 thru 0x3bbb43 (0x1a91 bytes) 000131[0] 1790100750.541521 write 0x091c81 thru 0x093499 (0x1819 bytes) 000132[1] 1790100750.560505 write 0x28554b thru 0x2874fc (0x1fb2 bytes) 000134[1] 1790100750.656224 trunc from 0x728001 to 0x4fe011 (0x229fef bytes) 000135[1] 1790100750.697298 punch from 0x145b98 to 0x1472b9 (0x1722 bytes) 000137[0] 1790100750.744079 read 0x0f6caf thru 0x0f7239 (0x58b bytes) 000138[0] 1790100750.775491 trunc from 0x4fe012 to 0x08fab0 (0x46e561 bytes) 000139[1] 1790100750.887549 write 0x539ab7 thru 0x53a96f (0xeb9 bytes) 000140[0] 1790100750.963564 write 0x23b91a thru 0x23d71c (0x1e03 bytes) 000143[1] 1790100751.054057 read 0x2bd8fa thru 0x2bde67 (0x56e bytes) 000144[0] 1790100751.080439 trunc from 0x53a970 to 0x1a6e8c (0x393ae3 bytes) 000145[1] 1790100751.157132 punch from 0x18dbc5 to 0x18e268 (0x6a4 bytes) 000146[0] 1790100751.232427 mapread 0x0ae924 thru 0x0b0172 (0x184f bytes) 000148[0] 1790100751.323686 punch from 0x0aee38 to 0x0af5c6 (0x78f bytes) 000149[0] 1790100751.411789 read 0x103988 thru 0x1042a8 (0x921 bytes) 000150[1] 1790100751.470629 mapwrite 0x655ea1 thru 0x656acc (0xc2c bytes) 000151[1] 1790100751.641188 write 0x0831d1 thru 0x083edb (0xd0b bytes) 000155[0] 1790100751.716704 mapread 0x2c8141 thru 0x2c9c11 (0x1ad1 bytes) 000156[0] 1790100751.860506 punch from 0x378a6f to 0x378df6 (0x388 bytes) 000159[0] 1790100751.931009 write 0x955401 thru 0x956e54 (0x1a54 bytes) 000160[1] 1790100751.949102 mapwrite 0x6eaab5 thru 0x6eafa4 (0x4f0 bytes) 000161[1] 1790100752.052793 punch from 0x10fe37 to 0x1112a0 (0x146a bytes) 000162[1] 1790100752.112027 punch from 0x4461cd to 0x447f4d (0x1d81 bytes) 000164[0] 1790100752.147085 write 0x1ba27f thru 0x1bbc88 (0x1a0a bytes) 000165[0] 1790100752.207393 punch from 0x764df7 to 0x766660 (0x186a bytes) 000166[1] 1790100752.301092 punch from 0x404dfe to 0x405318 (0x51b bytes) 000167[0] 1790100752.354858 mapread 0x5f3260 thru 0x5f3311 (0xb2 bytes) 000168[1] 1790100752.475800 mapread 0x39e5b9 thru 0x39f9e5 (0x142d bytes) 000169[0] 1790100752.519407 write 0x6949be thru 0x695bf6 (0x1239 bytes) 000171[1] 1790100752.558098 punch from 0x67e80b to 0x67f437 (0xc2d bytes) 000172[0] 1790100752.591501 punch from 0x543b92 to 0x54505e (0x14cd bytes) 000173[0] 1790100752.609677 mapwrite 0x838640 thru 0x839649 (0x100a bytes) 000177[1] 1790100752.884570 punch from 0x75abf2 to 0x75c8bf (0x1cce bytes) 000179[0] 1790100753.035935 mapwrite 0x39bea1 thru 0x39d0ef (0x124f bytes) 000180[1] 1790100753.164022 write 0x1ff276 thru 0x20009e (0xe29 bytes) 000181[0] 1790100753.272944 mapwrite 0x30a7b5 thru 0x30beb4 (0x1700 bytes) 000182[1] 1790100753.446998 write 0x539edc thru 0x53b338 (0x145d bytes) 000183[0] 1790100753.484944 trunc from 0x956e55 to 0x70ada8 (0x24c0ac bytes) 000185[1] 1790100753.630118 punch from 0x1347ac to 0x134b72 (0x3c7 bytes) 000189[1] 1790100753.762595 read 0x4f3124 thru 0x4f339a (0x277 bytes) 000190[0] 1790100753.865183 mapwrite 0x3c181f thru 0x3c22ab (0xa8d bytes) 000191[1] 1790100754.011952 mapread 0x26389b thru 0x2639cc (0x132 bytes) truncating to largest ever: 0x9dae14 000192[1] 1790100754.102034 trunc from 0x70ada9 to 0x9dae13 (0x2d006b bytes) 000195[1] 1790100754.295450 mapwrite 0x6f5025 thru 0x6f6d6e (0x1d4a bytes) 000196[1] 1790100754.364089 punch from 0x256ef9 to 0x2583b4 (0x14bc bytes) 000197[1] 1790100754.370126 mapread 0x48730b thru 0x488079 (0xd6f bytes) truncating to largest ever: 0x9e07eb 000199[0] 1790100754.411681 trunc from 0x9dae14 to 0x9e07ea (0x59d7 bytes) 000201[1] 1790100754.513814 trunc from 0x9e07eb to 0x3e647b (0x5fa36f bytes) 000202[1] 1790100754.598441 trunc from 0x3e647c to 0x2df633 (0x106e48 bytes) 000203[1] 1790100754.653661 mapwrite 0x85a57f thru 0x85bd61 (0x17e3 bytes) 000204[0] 1790100754.739111 punch from 0x350f1d to 0x35192c (0xa10 bytes) 000206[1] 1790100754.805435 mapread 0x1f35e4 thru 0x1f5587 (0x1fa4 bytes) 000207[1] 1790100754.879205 mapread 0x19f903 thru 0x1a1661 (0x1d5f bytes) 000208[1] 1790100754.891801 trunc from 0x85bd62 to 0x0edca9 (0x76e0b8 bytes) 000210[0] 1790100754.983839 read 0x06e666 thru 0x06f465 (0xe00 bytes) 000211[1] 1790100755.043063 trunc from 0x0edcaa to 0x021ec2 (0xcbde7 bytes) 000212[1] 1790100755.118025 trunc from 0x021ec3 to 0x5deee9 (0x5bd027 bytes) 000215[0] 1790100755.187487 write 0x6dc571 thru 0x6dc976 (0x406 bytes) 000217[1] 1790100755.232543 read 0x543244 thru 0x543d3c (0xaf9 bytes) 000218[0] 1790100755.275607 write 0x1d6bab thru 0x1d812f (0x1585 bytes) 000220[0] 1790100755.437959 read 0x137886 thru 0x13845d (0xbd8 bytes) 000221[1] 1790100755.463435 mapwrite 0x0e4006 thru 0x0e5f29 (0x1f24 bytes) 000222[1] 1790100755.559661 mapwrite 0x618aa1 thru 0x618c91 (0x1f1 bytes) 000223[1] 1790100755.748190 trunc from 0x6dc977 to 0x16561d (0x577359 bytes) 000225[1] 1790100755.780905 mapwrite 0x1d309e thru 0x1d350b (0x46e bytes) 000226[1] 1790100755.889826 mapwrite 0x1b8e32 thru 0x1b93f8 (0x5c7 bytes) 000227[1] 1790100755.983369 trunc from 0x1d350c to 0x5320c6 (0x35ebbb bytes) 000229[0] 1790100756.015067 mapread 0x1e2b93 thru 0x1e42a5 (0x1713 bytes) 000230[1] 1790100756.127004 write 0x6d754b thru 0x6d79d2 (0x488 bytes) 000231[0] 1790100756.154754 write 0x743787 thru 0x744e96 (0x1710 bytes) 000235[0] 1790100756.271086 punch from 0x3c33a3 to 0x3c389d (0x4fb bytes) 000236[0] 1790100756.297661 punch from 0x238045 to 0x239785 (0x1741 bytes) 000240[1] 1790100756.305962 read 0x014946 thru 0x015bcc (0x1287 bytes) 000241[0] 1790100756.363019 trunc from 0x744e97 to 0x5cfc44 (0x175252 bytes) 000243[1] 1790100756.424574 mapwrite 0x6967af thru 0x696ea1 (0x6f3 bytes) 000246[1] 1790100756.646794 mapread 0x646a2d thru 0x647319 (0x8ed bytes) 000247[1] 1790100756.684245 punch from 0x13445a to 0x135d02 (0x18a9 bytes) 000248[0] 1790100756.695884 punch from 0x316931 to 0x317a02 (0x10d2 bytes) 000249[1] 1790100756.752018 mapread 0x25c3e9 thru 0x25dbf3 (0x180b bytes) 000254[1] 1790100756.832937 mapread 0x2e94d7 thru 0x2e9830 (0x35a bytes) 000255[1] 1790100756.848397 mapwrite 0x2ab528 thru 0x2ac170 (0xc49 bytes) 000256[0] 1790100756.943647 write 0x4c06f6 thru 0x4c1ea5 (0x17b0 bytes) 000257[1] 1790100756.982678 write 0x728ca5 thru 0x7294ab (0x807 bytes) 000258[1] 1790100757.005865 read 0x6bdd71 thru 0x6be78a (0xa1a bytes) 000262[1] 1790100757.014038 write 0x0df7f6 thru 0x0e0969 (0x1174 bytes) 000264[1] 1790100757.025633 mapread 0x46a385 thru 0x46b799 (0x1415 bytes) 000265[0] 1790100757.058885 mapread 0x00df4b thru 0x00e87e (0x934 bytes) 000266[0] 1790100757.114780 write 0x266536 thru 0x267981 (0x144c bytes) 000268[1] 1790100757.207636 punch from 0x322ccb to 0x323b6e (0xea4 bytes) 000270[1] 1790100757.275890 mapread 0x2ad597 thru 0x2ad9de (0x448 bytes) 000271[1] 1790100757.283391 mapread 0x4970be thru 0x4989c6 (0x1909 bytes) 000273[0] 1790100757.297710 trunc from 0x7294ac to 0x76ff71 (0x46ac6 bytes) 000274[0] 1790100757.393120 punch from 0x720823 to 0x720af6 (0x2d4 bytes) 000275[1] 1790100757.403208 write 0x2eaa8a thru 0x2eba2c (0xfa3 bytes) 000276[0] 1790100757.444830 trunc from 0x76ff72 to 0x20950c (0x566a65 bytes) 000277[0] 1790100757.513878 punch from 0x157f97 to 0x1599a6 (0x1a10 bytes) 000278[0] 1790100757.523114 write 0x136882 thru 0x1386f7 (0x1e76 bytes) 000280[0] 1790100757.558359 mapwrite 0x29769f thru 0x297e64 (0x7c6 bytes) 000283[0] 1790100757.652014 read 0x243e57 thru 0x2445c7 (0x771 bytes) 000284[1] 1790100757.735586 mapwrite 0x722fb8 thru 0x723ffd (0x1046 bytes) 000285[1] 1790100757.848627 trunc from 0x723ffe to 0x73d8f4 (0x198f7 bytes) 000286[0] 1790100757.888762 read 0x43e228 thru 0x43fbb1 (0x198a bytes) 000287[0] 1790100757.922233 mapwrite 0x0e1de7 thru 0x0e220e (0x428 bytes) 000288[0] 1790100758.016545 mapwrite 0x2d18c9 thru 0x2d1f2b (0x663 bytes) 000289[0] 1790100758.063521 write 0x922542 thru 0x923f04 (0x19c3 bytes) 000290[1] 1790100758.079557 mapwrite 0x9a46e1 thru 0x9a5008 (0x928 bytes) 000292[1] 1790100758.206902 punch from 0x766a73 to 0x7685fa (0x1b88 bytes) 000294[0] 1790100758.265870 mapwrite 0x085208 thru 0x086e74 (0x1c6d bytes) 000295[1] 1790100758.443799 mapwrite 0x866ff7 thru 0x868757 (0x1761 bytes) 000296[0] 1790100758.548043 write 0x11b6bc thru 0x11bf60 (0x8a5 bytes) 000299[0] 1790100758.663037 write 0x80ca9f thru 0x80de08 (0x136a bytes) 000302[0] 1790100758.804539 mapread 0x1b7267 thru 0x1b861f (0x13b9 bytes) 000304[0] 1790100758.919130 mapwrite 0x4ca6dc thru 0x4cb165 (0xa8a bytes) 000305[0] 1790100759.161591 read 0x10503a thru 0x105f84 (0xf4b bytes) 000306[1] 1790100759.174728 trunc from 0x9a5009 to 0x556bf8 (0x44e410 bytes) 000307[0] 1790100759.280660 mapread 0x202bf8 thru 0x204399 (0x17a2 bytes) 000309[0] 1790100759.366596 mapwrite 0x9e1cec thru 0x9e36c4 (0x19d9 bytes) 000310[0] 1790100759.601428 mapwrite 0x8a86d3 thru 0x8a927f (0xbad bytes) 000311[1] 1790100759.774119 mapread 0x63f574 thru 0x63fff3 (0xa80 bytes) 000313[0] 1790100759.907972 mapread 0x05ba0c thru 0x05bc0b (0x200 bytes) 000315[1] 1790100759.949775 write 0x5e0a6e thru 0x5e0a74 (0x7 bytes) 000316[1] 1790100759.998038 read 0x12abb8 thru 0x12bf8a (0x13d3 bytes) 000318[1] 1790100760.114423 write 0x4d6ed1 thru 0x4d6fc2 (0xf2 bytes) 000321[0] 1790100760.157860 write 0x157333 thru 0x157521 (0x1ef bytes) 000322[0] 1790100760.206605 read 0x41d11b thru 0x41d257 (0x13d bytes) 000323[0] 1790100760.230700 punch from 0x4a0ed3 to 0x4a2d8f (0x1ebd bytes) 000324[1] 1790100760.273634 mapwrite 0x058617 thru 0x059c3a (0x1624 bytes) 000325[0] 1790100760.401911 mapread 0x9610e6 thru 0x96308b (0x1fa6 bytes) 000326[1] 1790100760.443548 write 0x425f83 thru 0x427d2d (0x1dab bytes) 000327[0] 1790100760.494998 mapread 0x053556 thru 0x05514e (0x1bf9 bytes) 000329[1] 1790100760.558328 write 0x1de460 thru 0x1dfb8b (0x172c bytes) 000332[0] 1790100760.633126 trunc from 0x9e36c5 to 0x801bb1 (0x1e1b13 bytes) 000333[0] 1790100760.732706 punch from 0x6a9192 to 0x6aa34b (0x11ba bytes) 000334[0] 1790100760.751509 write 0x9da904 thru 0x9dbff5 (0x16f2 bytes) 000335[1] 1790100760.785079 write 0x6662a5 thru 0x667251 (0xfad bytes) 000336[1] 1790100760.842225 mapwrite 0x8f63b6 thru 0x8f64ba (0x105 bytes) 000338[0] 1790100760.975188 mapwrite 0x6b5a06 thru 0x6b5d7e (0x379 bytes) 000339[1] 1790100761.104463 punch from 0x6a663f to 0x6a7cd7 (0x1699 bytes) 000342[1] 1790100761.197571 punch from 0x76f6d6 to 0x7715bf (0x1eea bytes) 000343[0] 1790100761.215841 trunc from 0x9dbff6 to 0x171f71 (0x86a084 bytes) 000344[0] 1790100761.314213 trunc from 0x171f72 to 0x965f7d (0x7f400c bytes) 000345[1] 1790100761.340925 trunc from 0x965f7e to 0x03617e (0x92fdff bytes) 000346[1] 1790100761.388377 punch from 0x01647e to 0x017fd3 (0x1b56 bytes) 000349[1] 1790100761.491167 mapwrite 0x3a771b thru 0x3a936a (0x1c50 bytes) 000351[1] 1790100761.675153 mapwrite 0x57149f thru 0x57270e (0x1270 bytes) 000352[1] 1790100761.856569 read 0x39acf2 thru 0x39bdc0 (0x10cf bytes) 000354[1] 1790100761.869628 write 0x981b39 thru 0x983054 (0x151c bytes) 000358[1] 1790100761.950254 read 0x51a98f thru 0x51bcdc (0x134e bytes) 000360[0] 1790100761.965528 mapwrite 0x9433b1 thru 0x943581 (0x1d1 bytes) 000366[1] 1790100762.287126 mapread 0x24abd6 thru 0x24bc3e (0x1069 bytes) 000367[0] 1790100762.340142 mapwrite 0x48ca90 thru 0x48dafd (0x106e bytes) 000369[1] 1790100762.463654 read 0x40a67f thru 0x40c548 (0x1eca bytes) 000370[0] 1790100762.498580 mapwrite 0x4c82b9 thru 0x4c9367 (0x10af bytes) 000371[0] 1790100762.679679 punch from 0x025463 to 0x026b62 (0x1700 bytes) 000375[0] 1790100762.778726 mapread 0x2c43fb thru 0x2c611a (0x1d20 bytes) 000377[0] 1790100762.954669 trunc from 0x983055 to 0x5a275e (0x3e08f6 bytes) 000380[0] 1790100763.044580 punch from 0x3d0a7b to 0x3d0b72 (0xf8 bytes) 000382[1] 1790100763.164906 trunc from 0x5a275f to 0x94b043 (0x3a88e5 bytes) 000384[1] 1790100763.352014 mapwrite 0x9b410c thru 0x9b5375 (0x126a bytes) 000385[1] 1790100763.514870 punch from 0x677d8e to 0x67903a (0x12ad bytes) 000386[1] 1790100763.526687 write 0x19964e thru 0x19a581 (0xf34 bytes) 000387[1] 1790100763.559874 mapwrite 0x9db509 thru 0x9dbaf2 (0x5ea bytes) 000388[1] 1790100763.636978 write 0x0b23fa thru 0x0b30da (0xce1 bytes) 000389[0] 1790100763.651700 mapwrite 0x47718a thru 0x47875a (0x15d1 bytes) 000392[1] 1790100763.814653 mapwrite 0x8dc5b6 thru 0x8ddd96 (0x17e1 bytes) 000393[1] 1790100763.970800 read 0x3c26d3 thru 0x3c4332 (0x1c60 bytes) 000394[1] 1790100764.037527 write 0x292157 thru 0x292912 (0x7bc bytes) 000395[1] 1790100764.105151 read 0x8d1755 thru 0x8d267f (0xf2b bytes) 000396[0] 1790100764.175531 mapwrite 0x5f3fd5 thru 0x5f5f8b (0x1fb7 bytes) 000398[0] 1790100764.409923 write 0x70629d thru 0x708054 (0x1db8 bytes) 000399[1] 1790100764.471394 mapread 0x180add thru 0x181d28 (0x124c bytes) 000400[1] 1790100764.586522 trunc from 0x9dbaf3 to 0x6032c3 (0x3d882f bytes) 000401[1] 1790100764.704586 mapread 0x1eef74 thru 0x1f087d (0x190a bytes) 000403[0] 1790100764.879553 punch from 0x007c77 to 0x0084f3 (0x87d bytes) 000404[0] 1790100764.944544 mapwrite 0x02acc0 thru 0x02beec (0x122d bytes) 000405[1] 1790100765.038691 read 0x17dc6a thru 0x17ec3b (0xfd2 bytes) 000407[0] 1790100765.090520 write 0x5a2f78 thru 0x5a2fd8 (0x61 bytes) 000409[0] 1790100765.118711 mapread 0x171ed7 thru 0x172d6c (0xe96 bytes) 000410[1] 1790100765.133214 read 0x3f7973 thru 0x3f81f5 (0x883 bytes) 000411[0] 1790100765.139867 mapwrite 0x98b073 thru 0x98bbdd (0xb6b bytes) 000413[1] 1790100765.288186 read 0x83d9ac thru 0x83e2b8 (0x90d bytes) 000415[0] 1790100765.376074 write 0x245564 thru 0x24615c (0xbf9 bytes) 000416[0] 1790100765.400661 punch from 0x2edbe0 to 0x2ee65d (0xa7e bytes) 000418[1] 1790100765.434566 punch from 0x630c39 to 0x631290 (0x658 bytes) 000423[0] 1790100765.569576 punch from 0x654b1a to 0x654dcf (0x2b6 bytes) 000424[0] 1790100765.593826 mapwrite 0x275b88 thru 0x2760f6 (0x56f bytes) 000425[0] 1790100765.674854 write 0x3bd16e thru 0x3bd4b5 (0x348 bytes) 000426[1] 1790100765.681935 read 0x4992cf thru 0x49ac65 (0x1997 bytes) 000428[1] 1790100765.740792 trunc from 0x98bbde to 0x82414e (0x167a8f bytes) 000430[0] 1790100765.886539 read 0x5d296e thru 0x5d3f02 (0x1595 bytes) 000433[0] 1790100766.011329 write 0x5e9cf1 thru 0x5e9fac (0x2bc bytes) 000434[0] 1790100766.099337 trunc from 0x82414f to 0x81e8f5 (0x5859 bytes) 000437[1] 1790100766.239232 punch from 0x58c8d0 to 0x58cf36 (0x667 bytes) 000438[0] 1790100766.321534 punch from 0x3cb107 to 0x3ccdbc (0x1cb6 bytes) 000439[0] 1790100766.375461 mapwrite 0x6b5583 thru 0x6b5b28 (0x5a6 bytes) 000440[0] 1790100766.507849 punch from 0x447f3e to 0x44993f (0x1a02 bytes) 000441[0] 1790100766.541484 punch from 0x4ce6f1 to 0x4cfc5c (0x156c bytes) 000445[1] 1790100766.626859 mapwrite 0x93cb8c thru 0x93de79 (0x12ee bytes) 000446[1] 1790100766.746178 trunc from 0x93de7a to 0x1d1a75 (0x76c404 bytes) 000448[1] 1790100766.804578 mapwrite 0x197b2f thru 0x198d25 (0x11f7 bytes) 000451[1] 1790100766.940067 read 0x1408e3 thru 0x1417c4 (0xee2 bytes) 000453[0] 1790100767.027353 trunc from 0x1d1a76 to 0x0311dd (0x1a0898 bytes) 000455[1] 1790100767.131889 write 0x1b7142 thru 0x1b8713 (0x15d2 bytes) 000457[0] 1790100767.213943 mapwrite 0x9743c7 thru 0x9759b9 (0x15f3 bytes) 000460[1] 1790100767.619178 punch from 0x84736b to 0x8474e3 (0x179 bytes) 000461[0] 1790100767.676615 trunc from 0x9759ba to 0x098e9e (0x8dcb1b bytes) 000462[1] 1790100767.745218 read 0x04a85f thru 0x04be06 (0x15a8 bytes) 000464[1] 1790100767.788890 trunc from 0x098e9f to 0x2ef31f (0x256481 bytes) 000467[0] 1790100768.025524 punch from 0x20a5dc to 0x20af30 (0x955 bytes) 000468[1] 1790100768.085736 mapread 0x25f377 thru 0x25fd56 (0x9e0 bytes) 000470[0] 1790100768.204851 mapwrite 0x700410 thru 0x700633 (0x224 bytes) 000471[1] 1790100768.431989 read 0x502a18 thru 0x50334f (0x938 bytes) 000472[1] 1790100768.524115 mapwrite 0x9a488a thru 0x9a61eb (0x1962 bytes) 000473[0] 1790100769.033700 write 0x1bf676 thru 0x1c0e72 (0x17fd bytes) 000474[1] 1790100769.118587 write 0x621062 thru 0x6214cc (0x46b bytes) 000479[1] 1790100769.262377 read 0x7ee519 thru 0x7ee94b (0x433 bytes) 000482[0] 1790100769.325789 trunc from 0x9a61ec to 0x1c02c3 (0x7e5f28 bytes) 000483[0] 1790100769.407391 mapread 0x1a36f9 thru 0x1a3925 (0x22d bytes) 000484[1] 1790100769.431164 read 0x0bafc1 thru 0x0bcf76 (0x1fb6 bytes) 000487[1] 1790100769.529363 punch from 0x1baab3 to 0x1bbc0d (0x115b bytes) 000489[1] 1790100769.579126 punch from 0x03d0dd to 0x03ef1b (0x1e3f bytes) 000492[0] 1790100769.637416 trunc from 0x1c02c4 to 0x0bffeb (0x1002d8 bytes) 000493[1] 1790100769.724686 punch from 0x09242b to 0x092f33 (0xb09 bytes) 000494[0] 1790100769.769477 write 0x5ddd6c thru 0x5de6e9 (0x97e bytes) 000496[1] 1790100769.825450 mapread 0x247e92 thru 0x249775 (0x18e4 bytes) 000498[1] 1790100769.949667 trunc from 0x5de6ea to 0x732e6b (0x154782 bytes) 000499[0] 1790100770.021383 read 0x70300c thru 0x703990 (0x985 bytes) All operations completed A-OK! PASS 16b (35s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:12:56 (1790100776) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:13:01 (1790100781) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.525015 s, 16.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.557599 s, 15.0 MB/s ldlm.namespaces.MGC192.168.204.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c6482e71800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8c64852b4800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8c64852b4800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8c64852b4800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.6112 s, 7.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.26924 s, 24.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.12506 s, 33.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.24227 s, 14.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.13765 s, 11.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.65986 s, 22.5 MB/s PASS 16d (61s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:14:02 (1790100842) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.448035 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112902 s, 9.3 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:14:10 (1790100850) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:14:38 (1790100878) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 14:15:05 (1790100905) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 14:15:12 (1790100912) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (9s) == sanityn test 16j: race dio with buffered i/o ========== 14:15:21 (1790100921) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.54745 s, 16.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49664 s, 17.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.95742 s, 8.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.86905 s, 14.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.44884 s, 10.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46084 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11048 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.4426 s, 10.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.4719 s, 10.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.61761 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.0118 s, 6.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.80714 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.14854 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.87853 s, 6.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.08198 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.62493 s, 10.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.99713 s, 8.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.21059 s, 8.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.51645 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (43s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:16:04 (1790100964) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Sep 22 14:16:57 EDT 2026 + fsx 2 done: rc=0 Tue Sep 22 14:16:57 EDT 2026 + fsx 3 done: rc=0 Tue Sep 22 14:16:57 EDT 2026 + fsx 4 done: rc=0 Tue Sep 22 14:16:57 EDT 2026 + fsx 5 done: rc=0 Tue Sep 22 14:16:57 EDT 2026 PASS 16k (61s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:17:05 (1790101025) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 14:17:15 (1790101035) mmap test1: basic mmap operation (PASS, 0.19571s) mmap test2: MAP_PRIVATE not write back (PASS, 0.13402s) mmap test3: concurrent mmap ops on two nodes (PASS, 15.29s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4804s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.19216s) mmap test6: check mmap write/read content on two nodes (PASS, 0.2358s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.2892s) mmap test11: multi-thread shared mmap (PASS, 1.4519s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1343s) PASS 18 (32s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:17:47 (1790101067) oleg435-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:17:51 (1790101071) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:17:59 (1790101079) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 14:18:06 (1790101086) now is 1790101089 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 new atime is 1790101151 PASS 23 (71s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:19:17 (1790101157) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234883 387 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:19:23 (1790101163) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:19:30 (1790101170) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:19:38 (1790101178) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 14:19:41 (1790101181) utime: good mknod times 1790101182 <= 1790101183 <= 1790101183 for /mnt/lustre/f26a utime: good mknod times 1790101182 <= 1790101183 <= 1790101183 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 14:19:48 (1790101188) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:19:57 (1790101197) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-22 14:19:58.000000000 -0400 Birth: 2026-09-22 14:19:58.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-22 14:19:58.000000000 -0400 Birth: 2026-09-22 14:19:58.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:20:05 (1790101205) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:20:13 (1790101213) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:20:20 (1790101220) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg435-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.135@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 14:20:39 (1790101239) -rw-r--r-- 1 root root 0 Sep 22 14:20 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:20:48 (1790101248) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1211: 43168 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:20:54 (1790101254) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 14:21:02 (1790101262) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 14:21:05 (1790101265) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.71639 s, 30.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08215 s, 48.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82976 s, 28.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.93486 s, 27.1 MB/s contention_seconds 4, contention_events 24 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.78511 s, 29.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.91497 s, 27.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08942 s, 48.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.14779 s, 45.7 MB/s PASS 32c (26s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:21:32 (1790101292) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:21:35 (1790101295) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:21:38 (1790101298) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:21:41 (1790101301) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 14:21:45 (1790101305) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c6482e71800.ost_server_uuid in IDLE state after 0 sec PASS 34 (146s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:24:11 (1790101451) fail_loc=0x80000317 Race attempt 0 Wait for 50475 50512 for 60 sec... - open/close 641 (time 1790101468.10 total 10.00 last 64.09) - open/close 1329 (time 1790101478.12 total 20.02 last 68.66) - open/close 1983 (time 1790101488.12 total 30.03 last 65.38) - open/close 2640 (time 1790101498.14 total 40.04 last 65.61) - open/close 3301 (time 1790101508.14 total 50.04 last 66.09) - open/close 3980 (time 1790101518.16 total 60.06 last 67.74) total: 4000 open/close in 60.41 seconds: 66.21 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1861: 50475 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:25:23 (1790101523) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg435-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32961 s, 39.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31216 s, 40.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6122 s, 32.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.6725 s, 31.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46117 s, 35.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7442432) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.76186 s, 29.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.46291 s, 35.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7444480) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3787 s, 38.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.33624 s, 39.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.07493 s, 48.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.937374 s, 55.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7493632) after_dd(7444480) after(7495680) PASS 36 (321s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:30:44 (1790101844) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 - create 1165 (time 1790101858.72 total 10.01 last 116.44) - create 2422 (time 1790101868.73 total 20.01 last 125.63) - create 3688 (time 1790101878.73 total 30.01 last 126.58) - create 5013 (time 1790101888.73 total 40.02 last 132.44) - create 6250 (time 1790101898.74 total 50.02 last 123.61) - create 7428 (time 1790101908.74 total 60.03 last 117.78) - create 8726 (time 1790101918.75 total 70.03 last 129.78) - create 9929 (time 1790101928.75 total 80.03 last 120.25) total: 10000 create in 80.48 seconds: 124.26 ops/second PASS 37 (92s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:32:17 (1790101937) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:32:24 (1790101944) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:32:32 (1790101952) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 14:32:41 (1790101961) fail_loc=0x411 fail_loc=0 PASS 39d (6s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:32:49 (1790101969) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:33:06 (1790101986) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (17s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:33:23 (1790102003) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:33:40 (1790102020) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:33:58 (1790102038) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:34:11 (1790102051) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:34:25 (1790102065) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (13s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 14:34:38 (1790102078) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 14:34:53 (1790102093) Conflict PASS 41e (14s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 14:35:07 (1790102107) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 14:35:22 (1790102122) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 14:35:35 (1790102135) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2433: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 14:35:50 (1790102150) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1109s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:54:20 (1790103260) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (16s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:54:37 (1790103277) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:54:50 (1790103290) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:55:05 (1790103305) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:55:20 (1790103320) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (16s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:55:36 (1790103336) Conflict PASS 42f (15s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:55:50 (1790103350) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:56:06 (1790103366) Conflict PASS 42h (18s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:56:23 (1790103383) PASS 43a (83s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:57:46 (1790103466) Conflict PASS 43b (17s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:58:03 (1790103483) Conflict PASS 43c (19s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:58:22 (1790103502) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (16s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:58:38 (1790103518) Conflict PASS 43e (16s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:58:54 (1790103534) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:59:09 (1790103549) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (15s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:59:24 (1790103564) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:59:39 (1790103579) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (4s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:59:43 (1790103583) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (140s) == sanityn test 43k: unlink vs create ==================== 15:02:03 (1790103723) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1311s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 15:23:54 (1790105034) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Sep 22 15:24:03 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 15:24:08 (1790105048) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 15:24:23 (1790105063) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 15:24:37 (1790105077) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 15:24:51 (1790105091) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 15:25:05 (1790105105) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 15:25:20 (1790105120) Conflict PASS 44g (21s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 15:25:40 (1790105140) Conflict PASS 44h (22s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 15:26:02 (1790105162) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 15:26:06 (1790105166) PASS 45a (139s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 15:28:25 (1790105305) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 15:28:40 (1790105320) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 15:28:56 (1790105336) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 15:29:12 (1790105352) Conflict PASS 45e (16s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 15:29:28 (1790105368) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (15s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 15:29:43 (1790105383) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 15:29:59 (1790105399) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 15:30:14 (1790105414) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 15:30:18 (1790105418) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1277s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 15:51:35 (1790106695) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (11s) == sanityn test 46b: pdirops: link vs create ======================================================================== 15:51:46 (1790106706) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 15:51:57 (1790106717) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 15:52:09 (1790106729) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 15:52:20 (1790106740) Conflict PASS 46e (11s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 15:52:31 (1790106751) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 15:52:43 (1790106763) Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 15:52:55 (1790106775) Conflict PASS 46h (13s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 15:53:08 (1790106788) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 15:53:11 (1790106791) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 15:53:14 (1790106794) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (2s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 15:53:17 (1790106797) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 15:53:19 (1790106799) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 15:53:22 (1790106802) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 15:53:25 (1790106805) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 15:53:28 (1790106808) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 15:53:31 (1790106811) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.106403 s, 96.2 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (11s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 15:53:42 (1790106822) 0+1 records in 0+1 records out 158 bytes copied, 0.0171286 s, 9.2 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3696: kill: (219018) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 15:53:50 (1790106830) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0143104 s, 71.6 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06801 s, 15.4 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 15:54:09 (1790106849) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0412832 s, 24.8 kB/s PASS 51c (11s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 15:54:20 (1790106860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582806 s, 18.0 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (7s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 15:54:27 (1790106867) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 15:54:35 (1790106875) llite.lustre-ffff8c6482e71800.xattr_cache=1 llite.lustre-ffff8c64852b4800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:36.000000000 -0400 Modify: 2026-09-22 15:54:36.000000000 -0400 Change: 2026-09-22 15:54:36.000000000 -0400 Birth: 2026-09-22 15:54:36.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:42.000000000 -0400 Modify: 2026-09-22 15:54:43.000000000 -0400 Change: 2026-09-22 15:54:43.000000000 -0400 Birth: 2026-09-22 15:54:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:48.000000000 -0400 Modify: 2026-09-22 15:54:48.000000000 -0400 Change: 2026-09-22 15:54:48.000000000 -0400 Birth: 2026-09-22 15:54:48.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:48.000000000 -0400 Modify: 2026-09-22 15:54:49.000000000 -0400 Change: 2026-09-22 15:54:49.000000000 -0400 Birth: 2026-09-22 15:54:48.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:55.000000000 -0400 Modify: 2026-09-22 15:54:56.000000000 -0400 Change: 2026-09-22 15:54:56.000000000 -0400 Birth: 2026-09-22 15:54:54.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 15:54:55.000000000 -0400 Modify: 2026-09-22 15:54:56.000000000 -0400 Change: 2026-09-22 15:54:56.000000000 -0400 Birth: 2026-09-22 15:54:54.000000000 -0400 PASS 54 (30s) == sanityn test 55a: rename vs unlink target dir ========= 15:55:05 (1790106905) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (11s) == sanityn test 55b: rename vs unlink source dir ========= 15:55:17 (1790106917) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 15:55:28 (1790106928) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 15:55:44 (1790106944) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (13s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 15:55:57 (1790106957) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 15:56:00 (1790106960) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (16s) == sanityn test 55g: rename: race with trylock =========== 15:56:16 (1790106976) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (21s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 15:56:38 (1790106998) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790106999 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.670759 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (96s) SKIP: sanityn test_56b skipping ALWAYS excluded test 56b == sanityn test 60: Verify data_version behaviour ======== 15:58:15 (1790107095) PASS 60 (6s) == sanityn test 70a: cd directory && rm directory ======== 15:58:21 (1790107101) PASS 70a (5s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 15:58:26 (1790107106) PASS 70b (5s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 15:58:31 (1790107111) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.009706 s, 6.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 15:58:34 (1790107114) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0136435 s, 4.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 15:58:36 (1790107116) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 15:58:38 (1790107118) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00754678 s, 8.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (2s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 15:58:40 (1790107120) llite.lustre-ffff8c6482e71800.xattr_cache=1 llite.lustre-ffff8c64852b4800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 15:58:44 (1790107124) llite.lustre-ffff8c6482e71800.xattr_cache=1 llite.lustre-ffff8c64852b4800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 15:58:48 (1790107128) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0122841 s, 83.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0148518 s, 68.9 kB/s 231367: get lock2 231368: get lock1 231367: done 231367: get lock1 231368: done 231368: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231368: cannot get lock2: Resource deadlock avoided 231368: exit rc=0 231367: done 231367: exit rc=0 PASS 74 (9s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 15:58:57 (1790107137) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0432699 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241781 s, 43.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4379: 231950 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 15:59:08 (1790107148) opening files: ........ closing files: ........ PASS 76 (57s) == sanityn test 77a: check FIFO NRS policy =============== 16:00:05 (1790107205) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.311938 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440326 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766849 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768939 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808341 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11797 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13137 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988907 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131773 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111703 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124476 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108964 s, 9.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810411 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251403 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541775 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558176 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767468 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815053 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546368 s, 19.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815536 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860102 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594437 s, 17.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726325 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298858 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860201 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304967 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605381 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818006 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765479 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820056 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286974 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319613 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112244 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=5 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (7s) == sanityn test 77b: check CRR-N NRS policy ============== 16:00:12 (1790107212) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.374715 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610937 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764759 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732729 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172815 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147848 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166767 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180293 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185973 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12242 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153922 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195747 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1781 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198546 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183144 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199526 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194467 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380781 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352631 s, 29.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709713 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728297 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782208 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308155 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299417 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886015 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398824 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439024 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882045 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502789 s, 20.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539028 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536938 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758996 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659741 s, 15.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.332629 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800729 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948308 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887496 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102006 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10744 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124137 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121654 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109919 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150493 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136723 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157221 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124924 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137188 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156598 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14502 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13487 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875357 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619983 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701172 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656152 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077755 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708161 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890134 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783461 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759043 s, 13.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725889 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812136 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081291 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822088 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931314 s, 11.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868413 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965505 s, 10.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (10s) == sanityn test 77c: check ORR NRS policy ================ 16:00:22 (1790107222) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.34057 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614931 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774132 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958339 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998021 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108303 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08872 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125185 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930109 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145409 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155956 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173463 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184105 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174709 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195375 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219958 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218835 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507864 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407868 s, 25.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221476 s, 47.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285599 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00811877 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928473 s, 113 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102283 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00561818 s, 187 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595291 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363128 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107178 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765113 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088393 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471791 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274296 s, 38.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194997 s, 53.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.406241 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508026 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797619 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512939 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056771 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071006 s, 14.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217191 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801052 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22275 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228797 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199006 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194392 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224755 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114165 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207312 s, 50.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241515 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260421 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228949 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208442 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544379 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427846 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00243999 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256774 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144644 s, 72.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447479 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00733353 s, 143 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575907 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188397 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00542096 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00728134 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00779316 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272152 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144479 s, 72.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.451589 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542494 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036833 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575644 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11035 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136261 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111867 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162406 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127768 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161577 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188735 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175423 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196258 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201483 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191978 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19126 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237939 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355012 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509826 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382594 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039041 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277972 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430965 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350589 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491078 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668011 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534925 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563578 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589405 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651871 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066517 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692094 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679092 s, 15.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (11s) == sanityn test 77d: check TRR nrs policy ================ 16:00:33 (1790107233) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.29565 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697729 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756094 s, 13.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159925 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17232 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194574 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196914 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211783 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203412 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19706 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18674 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239461 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228411 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216532 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233744 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24851 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267457 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305683 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394871 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254078 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681039 s, 15.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295474 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426515 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446036 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262229 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462024 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283525 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047793 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826047 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415595 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485022 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491181 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418516 s, 25.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.375729 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039946 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712021 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109583 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769285 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104331 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13096 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183715 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168782 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201755 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196879 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173817 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139756 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194455 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238574 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269655 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237295 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363105 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355609 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346171 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460456 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493398 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461583 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293987 s, 35.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044007 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044202 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547399 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070521 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497039 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040598 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588399 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945453 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578694 s, 18.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270902 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342306 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893152 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149038 s, 7.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173637 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964578 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154466 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170809 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164961 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147264 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158469 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1757 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124264 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203579 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130285 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161895 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167995 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039476 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991188 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371283 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133874 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297325 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155248 s, 67.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201824 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519226 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297945 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230805 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513268 s, 20.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270403 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599403 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327077 s, 321 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0051075 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00461775 s, 227 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (11s) == sanityn test 77e: check TBF NID nrs policy ============ 16:00:44 (1790107244) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.35@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.361155 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638581 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824622 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954817 s, 11.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097665 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880326 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158073 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200092 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190511 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205833 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206475 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231057 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193775 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25192 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236139 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258193 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31272 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181677 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195216 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183524 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13916 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193591 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145647 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193461 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206565 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153413 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160316 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200852 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178204 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161858 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16413 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219848 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185374 s, 5.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.37327 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418417 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101147 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148287 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152887 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154539 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121669 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154426 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171215 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158819 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164924 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129125 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182204 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116112 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140525 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15212 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194161 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153952 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147606 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146023 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152804 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152315 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167014 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162327 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153247 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181519 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165153 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16708 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180434 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170359 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182168 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169961 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175717 s, 6.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279172 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364147 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460726 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352798 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109112 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917057 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123994 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142159 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122053 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160102 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130018 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116907 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125453 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179542 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1817 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151701 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204917 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00606878 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034223 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388326 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434016 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403826 s, 26.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626958 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555203 s, 18.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547235 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056304 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492777 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561262 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433979 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664748 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510206 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735177 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738376 s, 14.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424076 s, 39.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332424 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583372 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586206 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124734 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151245 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161086 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162456 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177692 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169144 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160815 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187765 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148104 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173571 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191446 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193604 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209889 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513043 s, 20.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562265 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934313 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547085 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055433 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571411 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600668 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101874 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603074 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668881 s, 15.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373591 s, 28.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051545 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713395 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828332 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734614 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860289 s, 12.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (18s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:01:02 (1790107262) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.310556 s, 54.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462013 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628875 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510978 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113775 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132378 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138486 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152151 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169541 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132644 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130004 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144716 s, 7.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171298 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167547 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160063 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160384 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727381 s, 14.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275141 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239403 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033863 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265987 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294964 s, 35.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517523 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333493 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382336 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393101 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331986 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455974 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533844 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510477 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458075 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478761 s, 21.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057678 s, 18.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276697 s, 60.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561259 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446199 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052173 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15244 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155153 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134824 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18047 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195191 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202086 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197394 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214908 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220172 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190614 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252015 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253692 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268517 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00578226 s, 181 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525574 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457602 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477603 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051184 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048458 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558341 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570278 s, 18.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298295 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038594 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400123 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416161 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045588 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00613321 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402788 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016376 s, 64.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.323096 s, 51.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423161 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538243 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751445 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658364 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759138 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137261 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18772 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197647 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199114 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182081 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22592 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210961 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161518 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169457 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215861 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23937 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280578 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00384549 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345801 s, 30.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582521 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064376 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691157 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539653 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01121 s, 93.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054615 s, 19.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661268 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996477 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687216 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695269 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274249 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324911 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029308 s, 35.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.338373 s, 49.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054888 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546209 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459861 s, 22.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189268 s, 5.5 MB/srunning as uid/gid/euid/egid 500/500/500/500, groups: 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175402 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207145 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148089 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16245 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178817 s, 5.9 MB/s 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215982 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20393 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201166 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196511 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196984 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23788 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224834 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510438 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563384 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209691 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0067903 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059988 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450885 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462634 s, 227 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493794 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656813 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436004 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452741 s, 232 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011026 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749904 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563325 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728104 s, 14.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028788 s, 36.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (17s) == sanityn test 77g: Change TBF type directly ============ 16:01:19 (1790107279) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (7s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:01:26 (1790107286) oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 PASS 77h (8s) == sanityn test 77i: Change rank of TBF rule ============= 16:01:34 (1790107294) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (17s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:01:51 (1790107311) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.46863 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256986 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050092 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528304 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880595 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152672 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159148 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16858 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14844 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192199 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157337 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182431 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188735 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18008 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208091 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219638 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182711 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321178 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477359 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721116 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419327 s, 25.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365196 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412627 s, 25.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427245 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049856 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499555 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516369 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588386 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588869 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598146 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964822 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708148 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770135 s, 13.6 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.84461 s, 15.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5533 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (55s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:02:46 (1790107366) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31367 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631324 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108921 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115635 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893363 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144842 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043439 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170112 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161775 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685764 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17599 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17071 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114115 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134412 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129158 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220704 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270134 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513595 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606399 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585661 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063316 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618169 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585189 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741166 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607548 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793615 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673145 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789925 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669917 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071772 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062035 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744806 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822616 s, 12.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.903 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.981 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.37034 s, 12.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201991 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221145 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252537 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290446 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340559 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357694 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698986 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478076 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47559 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181881 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156618 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222044 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335587 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185903 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161633 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129749 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138189 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127034 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145208 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100639 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840851 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074258 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825755 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605281 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710641 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106444 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125154 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130892 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139146 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162517 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117811 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159903 s, 6.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.191 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0359 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (164s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 16:05:30 (1790107530) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35954 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135945 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128727 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186183 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198226 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325479 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367232 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480234 s, 2.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452173 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334977 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416344 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457018 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463562 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51483 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413093 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394027 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448834 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585337 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10216 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702105 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106025 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116576 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919353 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720142 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131786 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929438 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120306 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848848 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926181 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109152 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102892 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133492 s, 7.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.25 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0071 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.35822 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534734 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117832 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143023 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156183 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123241 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483662 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344547 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197866 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190473 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213192 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345362 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228869 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281497 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307788 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321202 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471657 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450573 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756678 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734615 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080007 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926445 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091505 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753697 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894609 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639754 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747884 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758375 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793269 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890876 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980695 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110489 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107314 s, 9.8 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9945 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9842 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (165s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 16:08:15 (1790107695) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.815621 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973318 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950699 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121667 s, 8.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230848 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273763 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243458 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280273 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292503 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341921 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37843 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355996 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376333 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282048 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398464 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321828 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420237 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632219 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116624 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126688 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125762 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120011 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655558 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722892 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750026 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796102 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122619 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172464 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165111 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196723 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211165 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954904 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100305 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4275 s, 8.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99555 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.492022 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515592 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063408 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810466 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970385 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656459 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093303 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141364 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596222 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12689 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117604 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178016 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175283 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18783 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131065 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116006 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175819 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046898 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749873 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483507 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105478 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642679 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703492 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772546 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845987 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840641 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907149 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151475 s, 69.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110809 s, 9.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996515 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999492 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137025 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16604 s, 6.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97643 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0057 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.785936 s, 21.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996321 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132426 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101708 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931174 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18898 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190356 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303981 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317792 s, 3.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23649 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227788 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35277 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233282 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284024 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390452 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274989 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454244 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550055 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681424 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591948 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773567 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106255 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104248 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581008 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580513 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937881 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931272 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965017 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986315 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134173 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11857 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994327 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100142 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3792 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.65959 s, 18.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.882107 s, 19.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964476 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106578 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150838 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266507 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308678 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34999 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379599 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449317 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326825 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499848 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493444 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393849 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544442 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536396 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.519071 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.579559 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359475 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983588 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598296 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112661 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434006 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692325 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697222 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108516 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525391 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011076 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686251 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015829 s, 66.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707062 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624916 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650788 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00781546 s, 134 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.1447 s, 8.0 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2343 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3883 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2548 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25007 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1098 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135045 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22609 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285138 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284462 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187134 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330398 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308947 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280624 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256367 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296869 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272161 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402961 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446257 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388919 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323035 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014745 s, 71.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386686 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226421 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892709 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950192 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153106 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172403 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157176 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135579 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975948 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172349 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166439 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180129 s, 58.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241038 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168268 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158941 s, 66.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.3509 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7284 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.902752 s, 18.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662224 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917343 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135714 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614289 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307406 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10482 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596783 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175307 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938812 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237876 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319731 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283797 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269718 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334342 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402196 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437891 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328046 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798788 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929484 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103283 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220034 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134219 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263157 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032456 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316315 s, 33.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113006 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378114 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381253 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809216 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623526 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110393 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717962 s, 14.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4896 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98825 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.887058 s, 18.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074569 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128776 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255461 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269061 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295601 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316619 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255178 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296188 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302785 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236878 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246811 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252997 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324192 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312497 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377589 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374532 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01034 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110768 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561861 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099806 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044542 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600485 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108077 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565755 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562661 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122566 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696688 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897312 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100573 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363813 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02542 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105635 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.8265 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1411 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8002 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0644 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (478s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 16:16:14 (1790108174) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.837632 s, 20.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394231 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455569 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298484 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824649 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115094 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826506 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771625 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185865 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147718 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126168 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197964 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152945 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175686 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184368 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250779 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296247 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109908 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109318 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118722 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082373 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841363 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802457 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798167 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918325 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885304 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938417 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837815 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128141 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080231 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131157 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164084 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120995 s, 8.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.03639 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94848 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (53s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 16:17:07 (1790108227) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561204 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074292 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782149 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137649 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18163 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15616 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20721 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213312 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236947 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166652 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177884 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23512 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283042 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228165 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269031 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210877 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210492 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859022 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889128 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735286 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115996 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786629 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815625 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803167 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110576 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093774 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915096 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101267 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103987 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161368 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107727 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111355 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112687 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.50045 s, 11.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9593 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (57s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 16:18:04 (1790108284) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.540633 s, 31.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875405 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115004 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114101 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245577 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181573 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277369 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238172 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163154 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265544 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237505 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267477 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210777 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349407 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293296 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322923 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335419 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524491 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330053 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677255 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748034 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414297 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818909 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848353 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790621 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055503 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639515 s, 16.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934769 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115429 s, 90.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104835 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590918 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764028 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648589 s, 162 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86094 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25361 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (51s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 16:18:55 (1790108335) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.811562 s, 20.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10645 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634956 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159915 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274726 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244057 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252139 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223697 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190163 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31072 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340346 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356691 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35498 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411729 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292856 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343907 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674526 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855917 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459283 s, 22.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473431 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674279 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633825 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885353 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787132 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805784 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670934 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887266 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071261 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864515 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878321 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150014 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11141 s, 9.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1781 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0457 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2903 s, 10.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0279 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (105s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 16:20:40 (1790108440) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.15288 s, 14.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719854 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149895 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205868 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250632 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164626 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230706 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27259 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826305 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284543 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305424 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337497 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34187 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28238 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314855 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169757 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178748 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509329 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062039 s, 16.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836162 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699755 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405383 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792509 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512456 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811182 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867547 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699273 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880933 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844045 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108042 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108596 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941905 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122678 s, 8.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8327 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0027 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (79s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:21:59 (1790108519) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.726562 s, 23.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553113 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113428 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167076 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058457 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15271 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210556 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208621 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178557 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220709 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197724 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274462 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262089 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290148 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271491 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264051 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c6482e71800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c6482e71800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335938 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844013 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341712 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723007 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785763 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929077 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146153 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929076 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11723 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771773 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156117 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121603 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903413 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134729 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117599 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402959 s, 260 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310363 s, 33.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4593 s, 10.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0083 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.828791 s, 20.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631406 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883159 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477634 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997274 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812415 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133485 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103785 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119478 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724683 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194705 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250392 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208467 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237868 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29004 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334393 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291359 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747509 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750322 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611489 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585756 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712257 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530086 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484603 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.051567 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615995 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672102 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694744 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873756 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714197 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718698 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109395 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945303 s, 11.1 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3406 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0062 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.7495 s, 8.2 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0238 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (153s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 16:24:33 (1790108673) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg435-client.virtnet /mnt/lustre (opts:) Stopping client oleg435-client.virtnet /mnt/lustre opts: Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg435-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg435-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg435-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg435-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70672/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg435-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: kmemleak disabled oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Starting client oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c648d59a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c64982b1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c648d59a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c64982b1800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.590857 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856228 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102875 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865666 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22125 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276061 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214654 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253149 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279264 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269535 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292441 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335618 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365058 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c648d59a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321096 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35059 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442663 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439383 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334861 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376461 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342198 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332501 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352166 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361989 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367176 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376138 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372003 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356684 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362838 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377481 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379419 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392343 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364854 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385002 s, 2.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.78845 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34976 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.759252 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723389 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599924 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990256 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186508 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643142 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962551 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10577 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283145 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295657 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330991 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312092 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325172 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353987 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334937 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39617 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231497 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c648d59a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19863 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182913 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183706 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17382 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193255 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193777 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205116 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228594 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213915 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221673 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216393 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22361 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226094 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210467 s, 5.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28689 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24392 s, 4.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=1 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.02324 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94236 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (223s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 16:28:16 (1790108896) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 16:28:33 (1790108913) On MGS 192.168.204.135, active = nodemap.active=1 On MGS 192.168.204.135, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567363 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531982 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450514 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702696 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176662 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193327 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212844 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259312 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249058 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238269 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257882 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264868 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262296 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27271 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261291 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338344 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345586 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544225 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376256 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540457 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599115 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459364 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489625 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638558 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553124 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669359 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108493 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080815 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869081 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880729 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907806 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978644 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962592 s, 10.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.42117 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.37453 s, 19.5 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.135, TBF.id = On MGS 192.168.204.135, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.745695 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132995 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984581 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100099 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156533 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154738 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118939 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163623 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192548 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175377 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170968 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203304 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179997 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20969 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117946 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21887 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197428 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c648d59a000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698982 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576121 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103812 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092771 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931556 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09756 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133455 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705791 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868734 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985407 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100126 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996837 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135211 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258527 s, 40.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093707 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108476 s, 9.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.17577 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.02734 s, 34.6 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg435-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 On MGS 192.168.204.135, active = nodemap.active=1 On MGS 192.168.204.135, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.490884 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546459 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717548 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179539 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153174 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129435 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145218 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157691 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193256 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151545 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199934 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175486 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234085 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231587 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223882 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231106 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25768 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347699 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609664 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259748 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714777 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463483 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330245 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625495 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573237 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479146 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502665 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561922 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595536 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690562 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685973 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723683 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862355 s, 12.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg435-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg435-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.01672 s, 13.1 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92432 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.135, TBF.id = On MGS 192.168.204.135, active = nodemap.active=0 PASS 77kj (169s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 16:31:22 (1790109082) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 16:31:32 (1790109092) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg435-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.89886 s, 269 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg435-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0536 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 16:32:29 (1790109149) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.43229 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831427 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800828 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121063 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133107 s, 7.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141236 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177926 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154037 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223118 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240545 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225662 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297051 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286919 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285236 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2689 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270147 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c648d59a000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325324 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298786 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107712 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139231 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817716 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121241 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0047294 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979926 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114565 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996904 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00843105 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163318 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791822 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114875 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370222 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372266 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183726 s, 5.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.47119 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.19506 s, 20.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.1757 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229888 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298069 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167565 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414489 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447907 s, 2.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8c648d59a000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436222 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52526 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.540989 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31697 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419194 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472088 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511568 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.627588 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.745581 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.740392 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.659544 s, 1.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8c648d59a000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288872 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233456 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178721 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233512 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247952 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222542 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227451 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345897 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202809 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227238 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228303 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264867 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249532 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331596 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451434 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287171 s, 3.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg435-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg435-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4238 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46701 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (101s) == sanityn test 77o: Changing rank should not panic ====== 16:34:10 (1790109250) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 16:34:23 (1790109263) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (123s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 16:36:26 (1790109386) mds.MDS.mdt.nrs_policies=tbf oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg435-client: oleg435-server: ssh exited with exit code 36 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg435-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg435-client: oleg435-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 16:37:04 (1790109424) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 272934 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 16:38:00 (1790109480) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (59s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 16:38:59 (1790109539) oleg435-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg435-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg435-client: oleg435-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (9s) == sanityn test 79: xattr: intent error ================== 16:39:08 (1790109548) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 16:39:09.000000000 -0400 Modify: 2026-09-22 16:39:09.000000000 -0400 Change: 2026-09-22 16:39:09.000000000 -0400 Birth: 2026-09-22 16:39:09.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 16:39:25 (1790109565) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 16:39:28 (1790109568) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 16:39:31 (1790109571) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (2s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 16:39:34 (1790109574) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 16:39:36 (1790109576) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 16:39:39 (1790109579) - open/close 710 (time 1790109594.11 total 10.01 last 70.92) - open/close 1371 (time 1790109604.11 total 20.01 last 66.08) - open/close 2084 (time 1790109614.11 total 30.02 last 71.26) total: 2500 open/close in 35.47 seconds: 70.49 ops/second snapshot_time 1790109698.304420069 secs.nsecs start_time 1790109581.681007151 secs.nsecs elapsed_time 116.623412918 secs.nsecs open 2505 samples [usecs] 127 16806 4425059 11821767301 close 2504 samples [usecs] 150 4471 1123777 822142033 mknod 2501 samples [usecs] 636 16667 3889891 9379898553 unlink 2500 samples [usecs] 835 56319 5431218 24956972042 mkdir 2 samples [usecs] 1132 3735 4867 15231649 rmdir 2 samples [usecs] 5943 13053 18996 205700058 rename 2500 samples [usecs] 587 48627 15108318 155519130090 getattr 3762 samples [usecs] 9 6753 254386 213688640 parallel_rename_file 2500 samples [usecs] 587 48627 15108318 155519130090 crossdir_rename 2500 samples [usecs] 587 48627 15108318 155519130090 readpage 14 samples [usecs] 4779 30039 194812 3451691366 PASS 81d (124s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 16:41:43 (1790109703) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7630 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 16:41:49 (1790109709) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 16:41:52 (1790109712) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 16:42:04 (1790109724) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790109725 Finishing test test1 at 1790109729 PASS 85 (10s) == sanityn test 90: open/create and unlink striped directory ========================================================== 16:42:14 (1790109734) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 16:42:17 (1790109737) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 16:42:20 (1790109740) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 16:42:23 (1790109743) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4778 0x12aa 0x240000400 1 4740 0x1284 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4779 0x12ab 0x240000400 1 4741 0x1285 0x280000400 rc1=0 and rc2=0 PASS 93 (14s) == sanityn test 94: signal vs CP callback race =========== 16:42:37 (1790109757) osc.lustre-OST0000-osc-ffff8c648d59a000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8c64982b1800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c648d59a000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8c64982b1800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.155318 s, 52.7 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98085 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6395: 289776 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0608704 s, 67.3 kB/s osc.lustre-OST0000-osc-ffff8c648d59a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8c64982b1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c648d59a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c64982b1800.idle_timeout=debug PASS 94 (14s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 16:42:51 (1790109771) llite.lustre-ffff8c648d59a000.fast_read=0 llite.lustre-ffff8c64982b1800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0447063 s, 46.9 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8c648d59a000.fast_read=1 llite.lustre-ffff8c64982b1800.fast_read=1 PASS 95a (18s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 16:43:10 (1790109790) llite.lustre-ffff8c648d59a000.fast_read=0 llite.lustre-ffff8c64982b1800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.015746 s, 780 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8c648d59a000.fast_read=1 llite.lustre-ffff8c64982b1800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 16:43:22 (1790109802) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 16:43:26 (1790109806) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00896234 s, 457 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 16:43:32 (1790109812) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176601 s, 11.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 16:43:39 (1790109819) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160968 s, 13.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (6s) == sanityn test 100e: DoM: read on open and file size ==== 16:43:46 (1790109826) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00888271 s, 1.4 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 16:43:52 (1790109832) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00876372 s, 467 kB/s PASS 101a (6s) == sanityn test 101b: Discard DoM data on rename ========= 16:43:59 (1790109839) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0142236 s, 288 kB/s PASS 101b (6s) == sanityn test 101c: Discard DoM data on close-unlink === 16:44:05 (1790109845) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0160182 s, 256 kB/s PASS 101c (9s) == sanityn test 102a: Test open by handle of unlinked file ========================================================== 16:44:14 (1790109854) file: /mnt/lustre/f102a.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 40 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf40:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102a (8s) == sanityn test 102b: open by handle of a dir inside an encrypted dir ========================================================== 16:44:22 (1790109862) 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) Starting client: oleg435-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock,test_dummy_encryption 192.168.204.135@tcp:/lustre /mnt/lustre2 file: /mnt/lustre/d102b.sanityn/encdir/sub fh_bytes: 32 fh_type: 151 fh_data: d3 0b 00 00 02 00 00 00 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd3:0x3:0x0] access via mount point '/mnt/lustre2' - OK check_fhandle_syscalls test Passed! 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,test_dummy_encryption=%s,test_dummy_encryption,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 PASS 102b (17s) == sanityn test 103: Test size correctness with lockahead ========================================================== 16:44:40 (1790109880) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790109883 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790109886 Finishing test test23 at 1790109886 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 16:44:52 (1790109892) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 16:44:56 (1790109896) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 16:45:21 (1790109921) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (8s) == sanityn test 106b: Glimpse RPCs test for statx ======== 16:45:30 (1790109930) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0349333 s, 29.3 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 16:45:33.000000000 -0400 Modify: 2026-09-22 16:45:33.000000000 -0400 Change: 2026-09-22 16:45:33.000000000 -0400 Birth: 2026-09-22 16:45:32.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238877265927 -rw-r--r-- 1 0 0 1790109932 1790109933 1790109933 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 11 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238877265927 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 16:45:33.000000000 -0400 Modify: 2026-09-22 16:45:33.000000000 -0400 Change: 2026-09-22 16:45:33.000000000 -0400 Birth: 2026-09-22 16:45:32.000000000 -0400 1790109933 1024 11 PASS 106b (15s) == sanityn test 106c: Verify statx attributes mask ======= 16:45:44 (1790109944) PASS 106c (11s) == sanityn test 107a: Basic grouplock conflict =========== 16:45:55 (1790109955) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.418584 s, 25.1 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 4788 0x12b4 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 16:46:06 (1790109966) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.459713 s, 22.8 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 4748 0x128c 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7630 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 16:46:21 (1790109981) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0100112 s, 999 kB/s llite.lustre-ffff8c648dc50800.hybrid_io=0 llite.lustre-ffff8c648dc53000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.85735 s, 1.7 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 16:46:35 (1790109995) 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 192.168.204.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg435-client.virtnet /mnt/lustre (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: kmemleak disabled oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre2 PASS 109 (48s) == sanityn test 110: do not grant another lock on resend ========================================================== 16:47:23 (1790110043) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238910820355 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 16:47:26.000000000 -0400 Modify: 2026-09-22 16:47:26.000000000 -0400 Change: 2026-09-22 16:47:26.000000000 -0400 Birth: 2026-09-22 16:47:25.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg435-client.virtnet oleg435-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 16:50:13 (1790110213) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 16:50:17 (1790110217) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 16:50:21 (1790110221) lustre-OST0000-osc-ffff8c64852b4000 active. lustre-OST0001-osc-ffff8c64852b4000 active. lustre-OST0000-osc-ffff8c64852b4000 active. lustre-OST0001-osc-ffff8c64852b4000 active. MGC192.168.204.135@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 16:50:30 (1790110230) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 16:50:35 (1790110235) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 16:50:40 (1790110240) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 16:50:43 (1790110243) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0138517 s, 1.5 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.141466 s, 145 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (11s) == sanityn test 122: directory size is consistent across mounts ========================================================== 16:50:54 (1790110254) PASS 122 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 16:51:06 (1790110266) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1119 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.149 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1228 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.137 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1273 s, 0.2 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0934 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.167 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0007 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1268 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0376 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1228 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1031 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2026 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1473 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1447 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2213 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1663 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2147 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg435-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8c64852b4000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c64852b4000.ost_server_uuid in FULL state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 10050 sec ============= 16:52:33 (1790110353) === sanityn: start cleanup 16:52:34 (1790110354) === Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 16:57:10 (1790110630) ===