-----============= acceptance-small: sanityn ============----- Mon Sep 14 02:16:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg110-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 02:17:15 (1789366635) === Starting client oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Mount client oleg110-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Started clients oleg110-client.virtnet: 192.168.201.110@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg110-client.virtnet: executing check_config_client /mnt/lustre oleg110-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg110-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d628319d000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d62a0d23000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d628319d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d62a0d23000.idle_timeout=debug disable quota as required oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 02:17:46 (1789366666) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337692 s, 31.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:17:49 (1789366669) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:17:56 (1789366676) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:18:03 (1789366683) -rw-r--r-- 1 root root 0 Sep 14 02:18 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:18:09 (1789366689) -rw-r--r-- 1 root root 0 Sep 14 02:18 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:18:15 (1789366695) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:18:21 (1789366701) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:18:27 (1789366707) total 1 -rwxrwxrwx 1 root root 0 Sep 14 02:18 f2a -rwxrwxrwx 1 root root 0 Sep 14 02:18 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 14 02:18 f2a -rwxrwxrwx 1 root root 0 Sep 14 02:18 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0177313 s, 28.9 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:18:33 (1789366713) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 02:18:36 (1789366716) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.352234 s, 6.0 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:18:43 (1789366723) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:18:49 (1789366729) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:18:56 (1789366736) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 02:19:03 (1789366743) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:19:09 (1789366749) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:19:16 (1789366756) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:19:23 (1789366763) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 02:19:30 (1789366770) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614573 s, 17.1 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0525164 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:19:37 (1789366777) 1+0 records in 1+0 records out 1 byte copied, 0.00399709 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102973 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00782166 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00896263 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00782658 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0132301 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0115381 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0118891 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0167776 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0188376 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129764 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0147474 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:19:45 (1789366785) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0127374 s, 241 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0480488 s, 85.2 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00379485 s, 810 kB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:19:52 (1789366792) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:19:59 (1789366799) warning: '-runtime' deprecated, use '-t runtime' instead - link 344 (time 1789366812.02 total 10.05 last 34.23) - link 663 (time 1789366822.02 total 20.05 last 31.89) - link 993 (time 1789366832.02 total 30.05 last 33.00) - link 1329 (time 1789366842.03 total 40.07 last 33.55) - link 1582 (time 1789366852.08 total 50.12 last 25.17) - link 1898 (time 1789366862.09 total 60.12 last 31.58) - link 2235 (time 1789366872.11 total 70.14 last 33.63) - link 2552 (time 1789366882.12 total 80.16 last 31.66) - link 2893 (time 1789366892.17 total 90.20 last 33.95) total: 3225 link in 100.01 seconds: 32.25 ops/second using seed 3400031821 running for 100 seconds - stat 5611 (time 1789366814 ; total 11 ; last 11) - stat 10000 (time 1789366822 ; total 19 ; last 8) - stat 15428 (time 1789366833 ; total 30 ; last 11) - stat 20000 (time 1789366841 ; total 38 ; last 8) - stat 24954 (time 1789366852 ; total 49 ; last 11) - stat 27888 (time 1789366863 ; total 60 ; last 11) - stat 30000 (time 1789366870 ; total 67 ; last 7) - stat 33183 (time 1789366881 ; total 78 ; last 11) - stat 36253 (time 1789366892 ; total 89 ; last 11) - stat 39190 (time 1789366903 ; total 100 ; last 11) total: 39191 stats in 100 seconds: 391.910004 stats/second - unlinked 0 (time 1789366853 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3225) error: No such file or directory total: 3224 unlinks in 86 seconds: 37.488373 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21415) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21417) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 02:22:24 (1789366944) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:22:34 (1789366954) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 22600 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:22:42 (1789366962) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 23180 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:22:49 (1789366969) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23759 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:22:57 (1789366977) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24341 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:23:05 (1789366985) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24923 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:23:14 (1789366994) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 02:24:06 (1789367046) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.510196 s, 16.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.57214 s, 14.7 MB/s Chance of close/open is 1 in 50 Seed set to 7051 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1789367052.183287 mapwrite 0x105b8a thru 0x114b68 (0xefdf bytes) 000003[1] 1789367052.414043 mapwrite 0x4891a1 thru 0x493a54 (0xa8b4 bytes) 000007[0] 1789367052.728734 write 0x102368 thru 0x10f87d (0xd516 bytes) truncating to largest ever: 0x12d351 000008[0] 1789367052.845653 trunc from 0x493a55 to 0x12d350 (0x366704 bytes) 000009[1] 1789367052.932379 punch from 0x098036 to 0x09bace (0x3a99 bytes) 000011[0] 1789367053.070751 mapread 0x0621ca thru 0x06c0b5 (0x9eec bytes) 000012[0] 1789367053.165359 mapread 0x0b426e thru 0x0c2ee9 (0xec7c bytes) 000015[0] 1789367053.229740 mapwrite 0x869e26 thru 0x87502e (0xb209 bytes) 000016[1] 1789367053.437061 mapread 0x6ff6b5 thru 0x704255 (0x4ba1 bytes) 000017[0] 1789367053.511725 mapread 0x260d9a thru 0x2631c6 (0x242d bytes) 000018[0] 1789367053.554332 punch from 0x5c0f41 to 0x5c6024 (0x50e4 bytes) 000020[1] 1789367053.628518 mapread 0x06eec3 thru 0x076f24 (0x8062 bytes) 000021[0] 1789367053.694697 write 0x03b565 thru 0x049de4 (0xe880 bytes) 000022[1] 1789367053.718925 punch from 0x150867 to 0x15410e (0x38a8 bytes) 000023[0] 1789367053.789529 punch from 0x04d3ff to 0x054783 (0x7385 bytes) 000024[0] 1789367053.810095 write 0x9392ba thru 0x948446 (0xf18d bytes) truncating to largest ever: 0x81e5f5 000025[1] 1789367053.811850 trunc from 0x948447 to 0x81e5f4 (0x129e52 bytes) 000026[1] 1789367053.876215 punch from 0x514a1b to 0x5173aa (0x2990 bytes) 000027[0] 1789367053.881577 read 0x7477b0 thru 0x74e000 (0x6851 bytes) 000029[1] 1789367053.934321 punch from 0x6fe6e4 to 0x70d27d (0xeb9a bytes) 000033[1] 1789367053.977368 read 0x2a28a9 thru 0x2b05d7 (0xdd2f bytes) 000034[1] 1789367053.988754 write 0x9df866 thru 0x9e682f (0x6fca bytes) 000035[1] 1789367054.023359 read 0x10bcde thru 0x1147e7 (0x8b0a bytes) 000036[0] 1789367054.035010 mapread 0x226026 thru 0x229d8c (0x3d67 bytes) 000037[1] 1789367054.117405 mapwrite 0x4b3fd0 thru 0x4b9fbb (0x5fec bytes) 000041[0] 1789367054.275529 mapread 0x474c74 thru 0x475c53 (0xfe0 bytes) 000043[0] 1789367054.314641 write 0x4f4f43 thru 0x4f7f2f (0x2fed bytes) 000044[0] 1789367054.360249 mapread 0x589174 thru 0x58d7e9 (0x4676 bytes) 000045[1] 1789367054.397638 mapread 0x92c466 thru 0x93a384 (0xdf1f bytes) 000047[0] 1789367054.446747 write 0x9dbae2 thru 0x9dc88c (0xdab bytes) 000048[1] 1789367054.473420 write 0x9a8037 thru 0x9ade28 (0x5df2 bytes) 000049[1] 1789367054.531176 mapread 0x88a20c thru 0x899558 (0xf34d bytes) truncating to largest ever: 0x8f1a60 000051[0] 1789367054.580025 trunc from 0x9e6830 to 0x8f1a5f (0xf4dd0 bytes) 000053[1] 1789367054.659682 read 0x3287e0 thru 0x33639d (0xdbbe bytes) 000054[1] 1789367054.679603 mapread 0x870d93 thru 0x874d15 (0x3f83 bytes) 000055[0] 1789367054.740491 mapwrite 0x4eb013 thru 0x4fab6c (0xfb5a bytes) 000057[1] 1789367054.818541 punch from 0x6cbf5e to 0x6d1c39 (0x5cdc bytes) 000059[1] 1789367054.876696 mapread 0x3d534b thru 0x3dd84d (0x8503 bytes) 000060[1] 1789367054.954113 read 0x61666d thru 0x625121 (0xeab5 bytes) 000062[1] 1789367054.968322 trunc from 0x8f1a60 to 0x49d0bf (0x4549a0 bytes) 000065[0] 1789367055.093622 mapwrite 0x9c2087 thru 0x9d1c57 (0xfbd1 bytes) 000067[1] 1789367055.348664 mapread 0x2c93a7 thru 0x2d32df (0x9f39 bytes) 000068[1] 1789367055.444503 punch from 0x65cc65 to 0x66587a (0x8c16 bytes) 000073[1] 1789367055.534655 mapwrite 0x9dedc8 thru 0x9e4b54 (0x5d8d bytes) 000075[0] 1789367055.711340 mapread 0x7f1f0d thru 0x7f50ea (0x31de bytes) 000078[0] 1789367055.816334 mapwrite 0x423882 thru 0x42c039 (0x87b8 bytes) 000081[1] 1789367056.066515 mapwrite 0x415d9a thru 0x417e73 (0x20da bytes) 000082[1] 1789367056.219396 read 0x554355 thru 0x5610ac (0xcd58 bytes) 000083[0] 1789367056.246349 write 0x1c7c06 thru 0x1cf432 (0x782d bytes) 000084[0] 1789367056.308280 mapread 0x73cc4c thru 0x73f603 (0x29b8 bytes) 000085[0] 1789367056.408574 read 0x8b3943 thru 0x8ba833 (0x6ef1 bytes) 000086[0] 1789367056.429086 mapwrite 0x25efc6 thru 0x265453 (0x648e bytes) 000087[1] 1789367056.561565 trunc from 0x9e4b55 to 0x867560 (0x17d5f4 bytes) 000089[1] 1789367056.629239 read 0x716aa0 thru 0x71a987 (0x3ee8 bytes) 000090[1] 1789367056.696504 write 0x68737c thru 0x6970c2 (0xfd47 bytes) 000091[0] 1789367056.708611 trunc from 0x867561 to 0x363f43 (0x50361d bytes) 000093[1] 1789367056.848623 write 0x1ba3fb thru 0x1c5877 (0xb47d bytes) 000094[0] 1789367056.902553 read 0x06a93a thru 0x072a2a (0x80f1 bytes) 000095[1] 1789367057.025532 write 0x6b476b thru 0x6bfd24 (0xb5ba bytes) 000096[1] 1789367057.038693 write 0x9ed2f5 thru 0x9f881e (0xb52a bytes) 000098[1] 1789367057.093844 mapread 0x5dd7ee thru 0x5e5928 (0x813b bytes) 000100[0] 1789367057.139549 mapread 0x14d227 thru 0x1513b0 (0x418a bytes) 000101[0] 1789367057.246359 write 0x6d3440 thru 0x6d4f97 (0x1b58 bytes) 000102[1] 1789367057.343169 trunc from 0x9f881f to 0x7dd1f8 (0x21b626 bytes) 000104[0] 1789367057.404493 write 0x702db2 thru 0x709b34 (0x6d83 bytes) 000105[0] 1789367057.461374 read 0x60800b thru 0x60a247 (0x223d bytes) 000106[1] 1789367057.486607 read 0x6374e8 thru 0x647045 (0xfb5e bytes) 000107[1] 1789367057.527078 punch from 0x5d8566 to 0x5df8f1 (0x738c bytes) 000108[0] 1789367057.580409 write 0x197e7a thru 0x19cb74 (0x4cfb bytes) 000109[0] 1789367057.626833 mapwrite 0x9d7139 thru 0x9de6dd (0x75a5 bytes) 000110[1] 1789367057.732720 read 0x9b5d0e thru 0x9bca3a (0x6d2d bytes) 000112[0] 1789367057.763674 trunc from 0x9de6de to 0x3de3fb (0x6002e2 bytes) 000113[0] 1789367057.782361 mapread 0x2b1d88 thru 0x2ba424 (0x869d bytes) 000116[1] 1789367057.857186 read 0x0af1a1 thru 0x0b8f25 (0x9d85 bytes) 000117[0] 1789367057.962605 trunc from 0x3de3fc to 0x0ec5ba (0x2f1e41 bytes) 000122[0] 1789367058.090561 punch from 0x0a89cf to 0x0b7e21 (0xf453 bytes) 000123[1] 1789367058.127082 write 0x98274f thru 0x98751b (0x4dcd bytes) 000124[0] 1789367058.174127 mapread 0x7718ad thru 0x7798b1 (0x8005 bytes) 000125[0] 1789367058.332549 write 0x8014d2 thru 0x80b748 (0xa277 bytes) 000127[1] 1789367058.367462 mapwrite 0x57fc9d thru 0x58f9de (0xfd42 bytes) 000128[0] 1789367058.845407 mapread 0x7f0253 thru 0x7f02ae (0x5c bytes) 000129[0] 1789367058.874921 read 0x6deacd thru 0x6ea8a4 (0xbdd8 bytes) 000131[1] 1789367058.971354 write 0x8e3362 thru 0x8ec24e (0x8eed bytes) 000134[1] 1789367059.058932 mapwrite 0x26aa6b thru 0x276251 (0xb7e7 bytes) 000137[0] 1789367059.313249 mapread 0x735111 thru 0x73c70a (0x75fa bytes) 000139[0] 1789367059.445873 trunc from 0x98751c to 0x43aa30 (0x54caeb bytes) 000143[0] 1789367059.522549 punch from 0x40d5eb to 0x41b139 (0xdb4f bytes) 000144[0] 1789367059.549707 mapread 0x0c95ae thru 0x0d1789 (0x81dc bytes) 000146[0] 1789367059.625317 punch from 0x0ce7e2 to 0x0d9bf5 (0xb414 bytes) 000147[1] 1789367059.654932 punch from 0x10c72d to 0x1101ed (0x3ac1 bytes) 000150[0] 1789367059.724162 mapread 0x0c4bd7 thru 0x0cde5f (0x9289 bytes) 000152[1] 1789367059.779238 trunc from 0x43aa31 to 0x4ea16a (0xaf73a bytes) 000153[0] 1789367059.830055 punch from 0x365932 to 0x37334e (0xda1d bytes) 000154[0] 1789367059.856624 trunc from 0x4ea16b to 0x6444e6 (0x15a37c bytes) 000156[1] 1789367059.900562 mapread 0x226339 thru 0x233d78 (0xda40 bytes) 000157[0] 1789367060.069230 read 0x1f986b thru 0x206771 (0xcf07 bytes) 000159[0] 1789367060.098989 mapread 0x11cd17 thru 0x124ce8 (0x7fd2 bytes) 000162[1] 1789367060.159091 write 0x1944f1 thru 0x19c3fb (0x7f0b bytes) 000163[1] 1789367060.218318 mapwrite 0x2d65e7 thru 0x2e43f9 (0xde13 bytes) 000165[0] 1789367060.405745 punch from 0x2d68bf to 0x2e1c80 (0xb3c2 bytes) 000166[1] 1789367060.450395 mapwrite 0x5f42ff thru 0x5febc5 (0xa8c7 bytes) 000167[1] 1789367060.615504 mapwrite 0x787b5b thru 0x787ce6 (0x18c bytes) 000168[1] 1789367060.762046 write 0x434ee3 thru 0x442a95 (0xdbb3 bytes) 000169[1] 1789367060.787144 write 0x725801 thru 0x729b5f (0x435f bytes) 000170[0] 1789367060.800413 mapwrite 0x670ca3 thru 0x678cc3 (0x8021 bytes) 000171[1] 1789367060.932826 mapwrite 0x9d117f thru 0x9d31ee (0x2070 bytes) 000173[0] 1789367061.157498 mapread 0x45767d thru 0x4674d0 (0xfe54 bytes) 000174[1] 1789367061.210294 punch from 0x660287 to 0x66ab47 (0xa8c1 bytes) 000175[0] 1789367061.228452 read 0x3f4a75 thru 0x3fef15 (0xa4a1 bytes) 000176[1] 1789367061.256062 write 0x6f01e4 thru 0x6fd12c (0xcf49 bytes) 000177[1] 1789367061.281287 write 0x898ecb thru 0x8a3d17 (0xae4d bytes) 000179[1] 1789367061.360584 write 0x0a9df2 thru 0x0aab20 (0xd2f bytes) 000181[0] 1789367061.419587 mapwrite 0x94ce66 thru 0x94e5b7 (0x1752 bytes) 000182[0] 1789367061.551567 write 0x1c40f2 thru 0x1c92b7 (0x51c6 bytes) 000184[1] 1789367061.631076 mapread 0x61c53a thru 0x62c080 (0xfb47 bytes) 000185[0] 1789367061.736710 trunc from 0x9d31ef to 0x51da56 (0x4b5798 bytes) 000186[1] 1789367061.803547 write 0x6a36aa thru 0x6a9f9d (0x68f4 bytes) 000187[0] 1789367061.829456 write 0x8b5630 thru 0x8c4c75 (0xf646 bytes) 000190[0] 1789367061.878060 read 0x7b4f40 thru 0x7b6237 (0x12f8 bytes) 000191[1] 1789367061.906334 mapread 0x7c0276 thru 0x7ceaa2 (0xe82d bytes) 000194[0] 1789367062.187033 trunc from 0x8c4c76 to 0x07447d (0x8507f8 bytes) 000195[0] 1789367062.213110 mapread 0x02ee37 thru 0x039f7b (0xb145 bytes) 000197[0] 1789367062.253609 write 0x8ed5df thru 0x8f0d24 (0x3746 bytes) 000198[1] 1789367062.275194 punch from 0x02de54 to 0x036f18 (0x90c5 bytes) 000199[0] 1789367062.330554 write 0x2bd20c thru 0x2c28f7 (0x56ec bytes) 000200[1] 1789367062.380406 punch from 0x093ff2 to 0x098fd6 (0x4fe5 bytes) 000202[1] 1789367062.444009 write 0x0f2fd5 thru 0x0fb39a (0x83c6 bytes) 000203[0] 1789367062.471754 write 0x006929 thru 0x012a7e (0xc156 bytes) 000204[0] 1789367062.522118 trunc from 0x8f0d25 to 0x7f3b7a (0xfd1aa bytes) 000207[0] 1789367062.550181 read 0x55dbea thru 0x56161f (0x3a36 bytes) 000209[0] 1789367062.558509 mapread 0x7c1ee2 thru 0x7cd8e4 (0xba03 bytes) 000210[1] 1789367062.605846 write 0x501256 thru 0x50be4c (0xabf7 bytes) 000211[1] 1789367062.675399 punch from 0x44c607 to 0x45352b (0x6f25 bytes) 000213[1] 1789367062.713194 mapread 0x0e11c2 thru 0x0eb25e (0xa09d bytes) 000215[1] 1789367062.821539 punch from 0x2767ff to 0x282dd0 (0xc5d2 bytes) 000217[0] 1789367062.865448 punch from 0x092db4 to 0x0988a1 (0x5aee bytes) 000219[0] 1789367062.918815 write 0x40e5cb thru 0x416ff9 (0x8a2f bytes) 000220[1] 1789367062.950868 mapread 0x60de18 thru 0x613c4a (0x5e33 bytes) 000221[0] 1789367063.042450 mapread 0x6ab11b thru 0x6b36c6 (0x85ac bytes) 000223[1] 1789367063.129102 read 0x298d44 thru 0x2a5cd6 (0xcf93 bytes) 000225[1] 1789367063.161351 mapread 0x2a093e thru 0x2a9a76 (0x9139 bytes) 000226[1] 1789367063.187121 write 0x57dd49 thru 0x58bd28 (0xdfe0 bytes) 000227[0] 1789367063.248168 read 0x055123 thru 0x057032 (0x1f10 bytes) 000228[0] 1789367063.308250 punch from 0x49899a to 0x49f5b2 (0x6c19 bytes) 000230[1] 1789367063.359364 punch from 0x27af72 to 0x28532f (0xa3be bytes) 000231[1] 1789367063.387107 mapread 0x6bef3f thru 0x6c807e (0x9140 bytes) 000232[0] 1789367063.429322 read 0x12b9a9 thru 0x133e2f (0x8487 bytes) 000233[0] 1789367063.463427 mapread 0x084b86 thru 0x0864ef (0x196a bytes) 000236[0] 1789367063.494929 mapread 0x61c5a6 thru 0x61e9b0 (0x240b bytes) 000237[1] 1789367063.532536 mapwrite 0x32b152 thru 0x32dd5c (0x2c0b bytes) 000238[1] 1789367063.706467 write 0x8056d1 thru 0x805a70 (0x3a0 bytes) 000239[1] 1789367063.730078 write 0x8ebc66 thru 0x8f4e2e (0x91c9 bytes) 000242[0] 1789367063.753442 mapread 0x20b964 thru 0x217c0b (0xc2a8 bytes) 000243[0] 1789367063.823591 punch from 0x336005 to 0x33a938 (0x4934 bytes) 000244[0] 1789367063.839258 mapwrite 0x545fa1 thru 0x55507a (0xf0da bytes) 000248[1] 1789367064.029515 punch from 0x85e7e9 to 0x86d913 (0xf12b bytes) 000249[1] 1789367064.072916 punch from 0x0ceb59 to 0x0dd97e (0xee26 bytes) 000252[1] 1789367064.137099 mapwrite 0x10005a thru 0x1007e4 (0x78b bytes) 000254[0] 1789367064.204941 punch from 0x5dca03 to 0x5df2e5 (0x28e3 bytes) 000256[1] 1789367064.243140 trunc from 0x8f4e2f to 0x275ad8 (0x67f356 bytes) 000257[1] 1789367064.280281 write 0x0186e7 thru 0x01b8c5 (0x31df bytes) 000258[0] 1789367064.305422 read 0x0edba8 thru 0x0f802b (0xa484 bytes) 000259[0] 1789367064.337057 mapread 0x01fa36 thru 0x027dc7 (0x8392 bytes) 000260[1] 1789367064.358180 mapwrite 0x03df3c thru 0x03e12d (0x1f2 bytes) 000262[0] 1789367064.440959 read 0x26afdf thru 0x2758c4 (0xa8e6 bytes) 000264[0] 1789367064.534092 mapwrite 0x3b97d0 thru 0x3bb2b8 (0x1ae9 bytes) 000265[1] 1789367064.679652 mapread 0x21cbea thru 0x21e1b8 (0x15cf bytes) 000266[1] 1789367064.762965 write 0x4e53be thru 0x4e5ef9 (0xb3c bytes) 000267[1] 1789367064.792952 read 0x4ad298 thru 0x4ad827 (0x590 bytes) 000268[0] 1789367064.797228 mapread 0x2321d7 thru 0x2366a2 (0x44cc bytes) 000270[0] 1789367064.831338 mapwrite 0x4a37d4 thru 0x4ac063 (0x8890 bytes) 000271[0] 1789367064.994125 punch from 0x0223b4 to 0x0254ec (0x3139 bytes) 000276[1] 1789367065.074025 write 0x8ad2a9 thru 0x8b3701 (0x6459 bytes) 000277[0] 1789367065.097599 mapwrite 0x466ef6 thru 0x475c1a (0xed25 bytes) 000278[1] 1789367065.208665 write 0x29344b thru 0x2971f8 (0x3dae bytes) 000282[1] 1789367065.281430 mapread 0x1e9882 thru 0x1f3d30 (0xa4af bytes) 000283[1] 1789367065.336423 write 0x2589ed thru 0x266592 (0xdba6 bytes) 000285[0] 1789367065.389427 mapwrite 0x84cd67 thru 0x858485 (0xb71f bytes) 000286[1] 1789367065.520196 trunc from 0x8b3702 to 0x564bff (0x34eb02 bytes) 000289[0] 1789367065.671040 read 0x042902 thru 0x048c4a (0x6349 bytes) 000290[0] 1789367065.711306 punch from 0x2c597d to 0x2cb0fa (0x577e bytes) 000292[0] 1789367065.777749 trunc from 0x564c00 to 0x7c49dc (0x25fddd bytes) 000293[1] 1789367065.851249 mapwrite 0x17cad9 thru 0x17fb94 (0x30bc bytes) 000294[0] 1789367065.988545 read 0x1df8dc thru 0x1e263c (0x2d61 bytes) 000295[1] 1789367066.038288 trunc from 0x7c49dd to 0x0685bf (0x75c41d bytes) 000297[0] 1789367066.135146 punch from 0x04c1de to 0x0514a9 (0x52cc bytes) 000298[0] 1789367066.173973 punch from 0x0227d4 to 0x023784 (0xfb1 bytes) 000299[0] 1789367066.200052 mapwrite 0x4caa3d thru 0x4d6d17 (0xc2db bytes) 000300[1] 1789367066.367432 mapwrite 0x1db754 thru 0x1e22d1 (0x6b7e bytes) 000302[1] 1789367066.536893 write 0x42b6f9 thru 0x4317e6 (0x60ee bytes) 000303[1] 1789367066.588404 read 0x2f85a8 thru 0x3000ac (0x7b05 bytes) 000305[0] 1789367066.605675 write 0x18a670 thru 0x191ba9 (0x753a bytes) 000306[0] 1789367066.668551 write 0x63f628 thru 0x640b27 (0x1500 bytes) 000307[0] 1789367066.709551 punch from 0x5ff157 to 0x60382e (0x46d8 bytes) 000308[0] 1789367066.730459 mapread 0x42401e thru 0x432549 (0xe52c bytes) 000309[0] 1789367066.811486 read 0x296ed3 thru 0x29b33f (0x446d bytes) 000311[0] 1789367066.877971 mapread 0x23e6f6 thru 0x24da98 (0xf3a3 bytes) 000312[1] 1789367066.937193 write 0x1bf1bc thru 0x1c71ac (0x7ff1 bytes) 000313[0] 1789367066.999171 trunc from 0x640b28 to 0x4ee095 (0x152a92 bytes) 000315[0] 1789367067.099511 write 0x0bf5fe thru 0x0c469d (0x50a0 bytes) 000316[0] 1789367067.126566 punch from 0x1e5b50 to 0x1ee65c (0x8b0d bytes) 000317[0] 1789367067.157008 mapwrite 0x081fd9 thru 0x08ed81 (0xcda9 bytes) 000318[0] 1789367067.265346 punch from 0x1ed462 to 0x1fbf32 (0xead1 bytes) 000319[1] 1789367067.284440 mapwrite 0x66506f thru 0x673877 (0xe809 bytes) 000325[1] 1789367067.511145 write 0x2484c8 thru 0x248533 (0x6c bytes) 000327[1] 1789367067.539520 trunc from 0x673878 to 0x400441 (0x273436 bytes) 000329[1] 1789367067.581910 trunc from 0x400442 to 0x4493ae (0x48f6d bytes) 000330[0] 1789367067.608291 punch from 0x1dc936 to 0x1e0216 (0x38e1 bytes) 000331[0] 1789367067.640650 trunc from 0x4493af to 0x373db3 (0xd55fb bytes) 000332[0] 1789367067.692141 punch from 0x1f2181 to 0x1f68f9 (0x4779 bytes) 000333[0] 1789367067.729560 mapwrite 0x3559e5 thru 0x355d97 (0x3b3 bytes) 000338[0] 1789367067.898758 mapread 0x14eff0 thru 0x15a387 (0xb398 bytes) 000339[0] 1789367067.934194 trunc from 0x373db4 to 0x58e3d7 (0x21a624 bytes) 000341[1] 1789367067.983425 trunc from 0x58e3d8 to 0x48f3c4 (0xff013 bytes) 000343[0] 1789367068.058150 mapwrite 0x5970f3 thru 0x5a64ab (0xf3b9 bytes) 000345[1] 1789367068.232310 read 0x517064 thru 0x524d3d (0xdcda bytes) 000346[0] 1789367068.265400 write 0x8a3599 thru 0x8ae3d2 (0xae3a bytes) 000348[1] 1789367068.288307 trunc from 0x8ae3d3 to 0x7f463c (0xb9d96 bytes) 000350[1] 1789367068.439589 mapwrite 0x91bc90 thru 0x91f752 (0x3ac3 bytes) 000352[1] 1789367068.581016 mapread 0x4ec560 thru 0x4f1cee (0x578f bytes) 000353[0] 1789367068.604128 punch from 0x5a801f to 0x5b62e4 (0xe2c6 bytes) 000354[1] 1789367068.633312 write 0x469675 thru 0x471528 (0x7eb4 bytes) 000355[1] 1789367068.683286 read 0x402ffb thru 0x412c69 (0xfc6f bytes) 000358[0] 1789367068.772361 punch from 0x8359b4 to 0x845451 (0xfa9e bytes) 000359[1] 1789367068.804216 read 0x107f8a thru 0x1102f8 (0x836f bytes) 000360[1] 1789367068.846091 mapread 0x8ee1a9 thru 0x8f4be6 (0x6a3e bytes) 000361[0] 1789367068.880596 trunc from 0x91f753 to 0x33d77f (0x5e1fd3 bytes) 000362[0] 1789367068.965910 read 0x1a97c6 thru 0x1aa725 (0xf60 bytes) 000363[0] 1789367069.050289 read 0x1d5022 thru 0x1e2cf7 (0xdcd6 bytes) 000365[1] 1789367069.077737 read 0x0122cc thru 0x0130a8 (0xddd bytes) 000366[0] 1789367069.152646 read 0x1f993b thru 0x201e78 (0x853e bytes) 000369[0] 1789367069.241119 mapwrite 0x462e83 thru 0x469944 (0x6ac2 bytes) 000370[0] 1789367069.381578 mapread 0x2c55bc thru 0x2c8e78 (0x38bd bytes) 000372[0] 1789367069.423278 write 0x25149f thru 0x251c19 (0x77b bytes) 000373[1] 1789367069.467242 mapread 0x1e6b1a thru 0x1ed067 (0x654e bytes) 000374[0] 1789367069.621698 trunc from 0x469945 to 0x820a42 (0x3b70fe bytes) 000376[1] 1789367069.707870 mapwrite 0x3c813a thru 0x3d616b (0xe032 bytes) 000379[1] 1789367069.910334 mapwrite 0x4e2372 thru 0x4ef158 (0xcde7 bytes) 000383[1] 1789367070.148546 mapread 0x78aadc thru 0x792abd (0x7fe2 bytes) 000384[1] 1789367070.150132 punch from 0x7b286d to 0x7b761d (0x4db1 bytes) 000385[0] 1789367070.185929 mapread 0x06e279 thru 0x073870 (0x55f8 bytes) 000386[1] 1789367070.257065 mapwrite 0x91ec37 thru 0x926243 (0x760d bytes) 000388[0] 1789367070.519675 mapwrite 0x6146a4 thru 0x615e68 (0x17c5 bytes) 000390[1] 1789367070.722525 trunc from 0x926244 to 0x0e94be (0x83cd85 bytes) 000391[0] 1789367070.815685 punch from 0x0643e6 to 0x06f536 (0xb151 bytes) 000393[0] 1789367070.892042 trunc from 0x0e94bf to 0x5858d1 (0x49c413 bytes) 000394[1] 1789367070.947296 read 0x556025 thru 0x565525 (0xf501 bytes) 000396[0] 1789367071.073147 mapread 0x412349 thru 0x4214af (0xf167 bytes) 000398[1] 1789367071.190741 mapwrite 0x85d30b thru 0x85e83e (0x1534 bytes) 000399[1] 1789367071.426418 mapwrite 0x3298c4 thru 0x32f89b (0x5fd8 bytes) 000400[0] 1789367071.495325 mapwrite 0x7620a1 thru 0x76c73c (0xa69c bytes) 000402[1] 1789367071.734018 punch from 0x0b7952 to 0x0b852a (0xbd9 bytes) 000403[0] 1789367071.758409 read 0x7f7960 thru 0x80565d (0xdcfe bytes) 000405[0] 1789367071.845766 write 0x2eb624 thru 0x2f6bf9 (0xb5d6 bytes) 000406[1] 1789367071.913679 mapwrite 0x5cad95 thru 0x5d6cb4 (0xbf20 bytes) 000407[1] 1789367072.090160 write 0x6f567f thru 0x6fba28 (0x63aa bytes) 000409[1] 1789367072.130159 mapwrite 0x6e570d thru 0x6eff0f (0xa803 bytes) 000410[0] 1789367072.298662 mapread 0x49767e thru 0x4a74b4 (0xfe37 bytes) 000411[1] 1789367072.357910 read 0x04691b thru 0x0483d0 (0x1ab6 bytes) 000412[0] 1789367072.383295 trunc from 0x85e83f to 0x38311e (0x4db720 bytes) 000413[0] 1789367072.440201 punch from 0x0fa986 to 0x103e68 (0x94e3 bytes) 000414[0] 1789367072.477753 punch from 0x1cd8f0 to 0x1d5b63 (0x8274 bytes) 000415[1] 1789367072.485735 read 0x0864f0 thru 0x08dfc4 (0x7ad5 bytes) 000417[0] 1789367072.629020 write 0x6fdf64 thru 0x70bd04 (0xdda1 bytes) 000420[1] 1789367072.708524 write 0x06de4c thru 0x07a0e2 (0xc297 bytes) 000421[1] 1789367072.778042 mapread 0x2a4f17 thru 0x2b117d (0xc267 bytes) 000422[0] 1789367072.867419 write 0x2564ff thru 0x2572fa (0xdfc bytes) 000429[1] 1789367073.038571 punch from 0x19d543 to 0x1ab10f (0xdbcd bytes) 000430[1] 1789367073.081768 mapread 0x156593 thru 0x1591f9 (0x2c67 bytes) 000431[1] 1789367073.126976 mapread 0x0f0147 thru 0x0f30ac (0x2f66 bytes) 000432[1] 1789367073.177730 punch from 0x646bc8 to 0x64b817 (0x4c50 bytes) 000433[0] 1789367073.214570 write 0x0b91c4 thru 0x0c5c1f (0xca5c bytes) 000435[0] 1789367073.278031 read 0x361fbb thru 0x362298 (0x2de bytes) 000436[0] 1789367073.302048 mapread 0x30f853 thru 0x3139c2 (0x4170 bytes) 000437[1] 1789367073.350039 write 0x7283be thru 0x7286ee (0x331 bytes) 000438[1] 1789367073.363479 trunc from 0x7286ef to 0x6c4eb6 (0x63838 bytes) 000439[1] 1789367073.412229 mapread 0x3646f7 thru 0x366ad6 (0x23e0 bytes) 000440[0] 1789367073.475235 mapwrite 0x50de9b thru 0x51c09b (0xe201 bytes) 000441[1] 1789367073.602458 mapwrite 0x7ecb10 thru 0x7f9bbc (0xd0ad bytes) 000442[1] 1789367073.798605 trunc from 0x7f9bbd to 0x65068b (0x1a9531 bytes) 000443[0] 1789367073.843606 mapwrite 0x216ae4 thru 0x21b823 (0x4d40 bytes) 000444[0] 1789367073.985749 mapwrite 0x5a3576 thru 0x5a434c (0xdd7 bytes) 000446[0] 1789367074.115161 mapread 0x5737fc thru 0x574bde (0x13e3 bytes) 000447[0] 1789367074.133372 mapwrite 0x97052e thru 0x97853a (0x800d bytes) 000449[0] 1789367074.279017 write 0x9aed6d thru 0x9bab40 (0xbdd4 bytes) 000451[1] 1789367074.288518 write 0x3a3ad1 thru 0x3a44d2 (0xa02 bytes) 000455[1] 1789367074.393229 write 0x933c4c thru 0x94203a (0xe3ef bytes) 000456[0] 1789367074.422203 punch from 0x3913e9 to 0x395268 (0x3e80 bytes) 000457[0] 1789367074.451956 mapread 0x7973d4 thru 0x79ad04 (0x3931 bytes) 000458[1] 1789367074.484188 punch from 0x6fe845 to 0x6ff2bb (0xa77 bytes) 000460[0] 1789367074.533264 punch from 0x885a5f to 0x885b61 (0x103 bytes) 000461[0] 1789367074.557460 mapwrite 0x685b3a thru 0x694f30 (0xf3f7 bytes) 000462[1] 1789367074.653932 mapwrite 0x803bbb thru 0x808886 (0x4ccc bytes) 000463[1] 1789367074.784936 punch from 0x5f8791 to 0x5f9f0c (0x177c bytes) 000464[1] 1789367074.843441 punch from 0x46c9c2 to 0x46c9f3 (0x32 bytes) 000465[1] 1789367074.853640 mapread 0x292591 thru 0x2a2185 (0xfbf5 bytes) 000466[1] 1789367074.962155 read 0x869a61 thru 0x8770f3 (0xd693 bytes) 000467[1] 1789367074.987182 trunc from 0x9bab41 to 0x84e087 (0x16cab9 bytes) 000468[0] 1789367075.006911 trunc from 0x84e088 to 0x30644b (0x547c3c bytes) 000469[1] 1789367075.034753 write 0x4491c3 thru 0x44d921 (0x475f bytes) 000473[1] 1789367075.077719 read 0x410e76 thru 0x41c67b (0xb806 bytes) 000475[0] 1789367075.105124 punch from 0x081ab2 to 0x08e70a (0xcc59 bytes) 000478[1] 1789367075.142736 read 0x3e7e08 thru 0x3f6bf6 (0xedef bytes) 000479[0] 1789367075.214040 punch from 0x0142da to 0x01dee0 (0x9c07 bytes) 000481[1] 1789367075.265818 mapwrite 0x0ec65d thru 0x0f4ae9 (0x848d bytes) 000482[0] 1789367075.358705 read 0x2ce410 thru 0x2d64cd (0x80be bytes) 000483[1] 1789367075.394170 read 0x42e6ad thru 0x43091f (0x2273 bytes) 000485[1] 1789367075.433382 trunc from 0x44d922 to 0x0c485c (0x3890c5 bytes) 000487[1] 1789367075.488481 punch from 0x058411 to 0x0677a8 (0xf398 bytes) 000488[0] 1789367075.494435 punch from 0x00a9c0 to 0x01855c (0xdb9d bytes) 000489[1] 1789367075.541046 punch from 0x0b27ea to 0x0b7f6b (0x5782 bytes) 000490[0] 1789367075.599818 mapwrite 0x1155d0 thru 0x11c204 (0x6c35 bytes) 000491[1] 1789367075.901365 write 0x214b2f thru 0x22478c (0xfc5e bytes) 000493[1] 1789367075.968305 mapread 0x1bb478 thru 0x1bf6bb (0x4244 bytes) 000494[1] 1789367076.017152 mapread 0x1e374c thru 0x1e7b8b (0x4440 bytes) 000496[0] 1789367076.061389 read 0x181c07 thru 0x18f26a (0xd664 bytes) 000499[0] 1789367076.185969 mapread 0x0d1788 thru 0x0deb42 (0xd3bb bytes) 000500[0] 1789367076.187370 write 0x458b25 thru 0x463f71 (0xb44d bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7076 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero length punch hole truncating to largest ever: 0x76adee 000006[0] 1789367077.035984 trunc from 00000000 to 0x76aded (0x76adee bytes) 000009[0] 1789367077.107338 trunc from 0x76adee to 0x15d5c5 (0x60d828 bytes) 000011[1] 1789367077.131972 mapwrite 0x256000 thru 0x25eabe (0x8abf bytes) 000013[0] 1789367077.268239 punch from 0x09f945 to 0x0a0f5e (0x161a bytes) 000016[1] 1789367077.301071 mapwrite 0x1b7000 thru 0x1ba28e (0x328f bytes) 000017[1] 1789367077.385223 write_OD 0x75b000 thru 0x764fff (0xa000 bytes) 000018[1] 1789367077.440562 read_OD 0x171000 thru 0x17cfff (0xc000 bytes) 000019[0] 1789367077.456143 punch from 0x70dff6 to 0x715f40 (0x7f4b bytes) 000021[0] 1789367077.484760 mapread 0x1fa000 thru 0x1fd367 (0x3368 bytes) 000022[1] 1789367077.512101 mapwrite 0x146000 thru 0x151aa2 (0xbaa3 bytes) 000024[1] 1789367077.629840 mapread 0x424000 thru 0x429320 (0x5321 bytes) 000025[0] 1789367077.667090 mapread 0x23a000 thru 0x23d73b (0x373c bytes) 000026[0] 1789367077.712853 mapread 0x4c1000 thru 0x4cf483 (0xe484 bytes) 000027[1] 1789367077.736182 read_OD 0x2f8000 thru 0x301fff (0xa000 bytes) 000031[0] 1789367077.751965 mapwrite 0x034000 thru 0x034f8c (0xf8d bytes) 000032[0] 1789367077.852266 mapread 0x60a000 thru 0x60a96f (0x970 bytes) 000036[0] 1789367077.893559 punch from 0x67873d to 0x684980 (0xc244 bytes) 000038[1] 1789367077.903069 punch from 0x2cad5b to 0x2d1894 (0x6b3a bytes) 000039[0] 1789367077.922558 mapwrite 0x651000 thru 0x65544c (0x444d bytes) 000040[0] 1789367078.019662 mapwrite 0x87e000 thru 0x88d7b6 (0xf7b7 bytes) 000041[1] 1789367078.114388 mapread 0x2eb000 thru 0x2ee369 (0x336a bytes) 000043[1] 1789367078.163571 write_OD 0x6a5000 thru 0x6acfff (0x8000 bytes) truncating to largest ever: 0x7afe75 000045[0] 1789367078.233916 trunc from 0x88d7b7 to 0x7afe74 (0xdd942 bytes) 000046[0] 1789367078.286352 mapwrite 0x230000 thru 0x232857 (0x2858 bytes) 000047[0] 1789367078.383167 trunc from 0x7afe75 to 0x6a9909 (0x10656b bytes) 000050[0] 1789367078.442824 write 0x4e4000 thru 0x4e8ea2 (0x4ea3 bytes) 000051[1] 1789367078.459715 trunc from 0x6a990a to 0x764fad (0xbb6a4 bytes) 000052[1] 1789367078.507732 mapread 0x4f7000 thru 0x502353 (0xb354 bytes) 000053[0] 1789367078.567127 mapread 0x407000 thru 0x40c969 (0x596a bytes) 000054[0] 1789367078.612150 mapread 0x286000 thru 0x2943a3 (0xe3a4 bytes) 000058[1] 1789367078.665664 mapwrite 0x0cb000 thru 0x0cc13b (0x113c bytes) 000063[1] 1789367078.729114 mapread 0x021000 thru 0x02d556 (0xc557 bytes) truncating to largest ever: 0x91b36d 000064[1] 1789367078.781147 trunc from 0x764fae to 0x91b36c (0x1b63bf bytes) 000065[0] 1789367078.806913 write 0x9c7000 thru 0x9cc69b (0x569c bytes) 000066[0] 1789367078.824991 write 0x819000 thru 0x828c7a (0xfc7b bytes) 000069[1] 1789367078.886485 read_OD 0x371000 thru 0x372fff (0x2000 bytes) 000070[1] 1789367078.929830 mapread 0x60f000 thru 0x618f72 (0x9f73 bytes) 000072[0] 1789367078.962270 write 0x2a8000 thru 0x2aaeea (0x2eeb bytes) 000075[0] 1789367079.002628 punch from 0x41e322 to 0x4266c0 (0x839f bytes) 000077[1] 1789367079.028351 mapread 0x81a000 thru 0x81ecbf (0x4cc0 bytes) 000080[1] 1789367079.069864 mapread 0x756000 thru 0x760a22 (0xaa23 bytes) 000081[1] 1789367079.087876 write_OD 0x950000 thru 0x95cfff (0xd000 bytes) 000084[1] 1789367079.145564 punch from 0x33d723 to 0x347d7e (0xa65c bytes) 000087[1] 1789367079.194406 write_OD 0x8cb000 thru 0x8d8fff (0xe000 bytes) 000088[0] 1789367079.257616 write 0x2cc000 thru 0x2db903 (0xf904 bytes) 000089[1] 1789367079.288978 punch from 0x832caf to 0x834cf1 (0x2043 bytes) 000090[1] 1789367079.313101 punch from 0x58eb21 to 0x59d607 (0xeae7 bytes) 000091[0] 1789367079.334770 punch from 0x9a77cb to 0x9b3bc7 (0xc3fd bytes) 000094[1] 1789367079.359553 trunc from 0x9cc69c to 0x0dfd01 (0x8ec99a bytes) 000095[1] 1789367079.410044 mapwrite 0x87d000 thru 0x88a3b3 (0xd3b4 bytes) 000096[0] 1789367079.602916 mapwrite 0x413000 thru 0x41769d (0x469e bytes) 000097[0] 1789367079.707465 write 0x102000 thru 0x10fff2 (0xdff3 bytes) 000101[1] 1789367079.775742 read_OD 0x1c1000 thru 0x1c7fff (0x7000 bytes) 000103[1] 1789367079.839007 write_OD 0x1ed000 thru 0x1f5fff (0x9000 bytes) 000104[1] 1789367079.894720 trunc from 0x88a3b4 to 0x59924f (0x2f1164 bytes) 000105[1] 1789367079.949680 trunc from 0x599250 to 0x3e29fa (0x1b6855 bytes) 000109[0] 1789367080.090182 write 0x08d000 thru 0x09943b (0xc43c bytes) 000112[1] 1789367080.162447 trunc from 0x3e29fb to 0x637739 (0x254d3f bytes) 000113[0] 1789367080.251742 trunc from 0x63773a to 0x6185d1 (0x1f168 bytes) truncating to largest ever: 0x9d4019 000114[0] 1789367080.284871 trunc from 0x6185d2 to 0x9d4018 (0x3bba47 bytes) 000115[0] 1789367080.315466 punch from 0x3aa373 to 0x3af6d6 (0x5364 bytes) 000120[1] 1789367080.323626 trunc from 0x9d4019 to 0x1480e4 (0x88bf34 bytes) 000121[1] 1789367080.361154 mapwrite 0x5b6000 thru 0x5bb031 (0x5032 bytes) 000122[1] 1789367080.468524 trunc from 0x5bb032 to 0x5c3d40 (0x8d0f bytes) 000123[1] 1789367080.546046 mapwrite 0x65e000 thru 0x665fd3 (0x7fd4 bytes) 000125[0] 1789367080.692854 write 0x375000 thru 0x380507 (0xb508 bytes) 000133[1] 1789367080.788003 mapwrite 0x6ae000 thru 0x6b8e7a (0xae7b bytes) 000134[0] 1789367081.021237 punch from 0x0d71dc to 0x0dc73a (0x555f bytes) 000136[0] 1789367081.058471 trunc from 0x6b8e7b to 0x6e3afe (0x2ac84 bytes) 000138[1] 1789367081.124024 mapread 0x039000 thru 0x04400a (0xb00b bytes) 000139[1] 1789367081.188766 write_OD 0x70e000 thru 0x717fff (0xa000 bytes) 000140[1] 1789367081.246132 mapread 0x09c000 thru 0x09d895 (0x1896 bytes) 000142[0] 1789367081.262301 punch from 0x10648a to 0x109b9e (0x3715 bytes) 000144[1] 1789367081.317431 punch from 0x4cecf6 to 0x4dc5ba (0xd8c5 bytes) 000146[0] 1789367081.345476 write 0x085000 thru 0x085d2e (0xd2f bytes) 000148[0] 1789367081.397756 write 0x718000 thru 0x71e1c3 (0x61c4 bytes) 000149[1] 1789367081.432193 mapread 0x361000 thru 0x36c264 (0xb265 bytes) 000151[0] 1789367081.543364 mapread 0x651000 thru 0x65bdc3 (0xadc4 bytes) 000152[0] 1789367081.614675 trunc from 0x71e1c4 to 0x942a7f (0x2248bc bytes) 000153[0] 1789367081.687309 punch from 0x51a236 to 0x525521 (0xb2ec bytes) 000154[0] 1789367081.697261 mapwrite 0x9ab000 thru 0x9b93a3 (0xe3a4 bytes) 000157[1] 1789367081.829314 mapwrite 0x593000 thru 0x597bd9 (0x4bda bytes) 000158[0] 1789367081.954428 write 0x11b000 thru 0x128109 (0xd10a bytes) 000159[0] 1789367081.974161 mapwrite 0x612000 thru 0x615c78 (0x3c79 bytes) 000161[0] 1789367082.048419 mapread 0x019000 thru 0x01f13b (0x613c bytes) 000163[0] 1789367082.081071 mapread 0x945000 thru 0x9463a9 (0x13aa bytes) 000165[0] 1789367082.109331 read 0x98c000 thru 0x991ac4 (0x5ac5 bytes) 000166[1] 1789367082.134763 mapread 0x875000 thru 0x8775b1 (0x25b2 bytes) 000169[0] 1789367082.180974 mapwrite 0x3a9000 thru 0x3b3efc (0xaefd bytes) 000171[1] 1789367082.308373 trunc from 0x9b93a4 to 0x45d94f (0x55ba54 bytes) 000172[0] 1789367082.366259 punch from 0x0d1e40 to 0x0d2cff (0xec0 bytes) 000173[0] 1789367082.388314 trunc from 0x45d950 to 0x1697ae (0x2f41a1 bytes) 000174[1] 1789367082.408616 trunc from 0x1697af to 0x6f03e0 (0x586c32 bytes) 000175[1] 1789367082.460054 mapwrite 0x0c7000 thru 0x0d4737 (0xd738 bytes) 000176[1] 1789367082.549449 trunc from 0x6f03e1 to 0x93f291 (0x24eeb1 bytes) 000178[0] 1789367082.595050 trunc from 0x93f292 to 0x14cdce (0x7f24c3 bytes) 000179[1] 1789367082.653635 write_OD 0x390000 thru 0x398fff (0x9000 bytes) 000180[0] 1789367082.732490 mapwrite 0x331000 thru 0x3310da (0xdb bytes) 000181[0] 1789367082.810943 read 0x102000 thru 0x110fb6 (0xefb7 bytes) 000183[0] 1789367082.919414 write 0x971000 thru 0x972b13 (0x1b14 bytes) 000186[0] 1789367082.939308 trunc from 0x972b14 to 0x460515 (0x5125fe bytes) 000187[0] 1789367082.960532 mapwrite 0x90d000 thru 0x91a72e (0xd72f bytes) 000191[0] 1789367083.161949 mapwrite 0x403000 thru 0x41204c (0xf04d bytes) 000193[0] 1789367083.245664 trunc from 0x91a72f to 0x705b22 (0x214c0c bytes) 000194[0] 1789367083.329424 trunc from 0x705b23 to 0x72f784 (0x29c62 bytes) 000195[1] 1789367083.361273 punch from 0x4950df to 0x4955f5 (0x517 bytes) 000196[0] 1789367083.417554 write 0x49c000 thru 0x4a84a7 (0xc4a8 bytes) 000197[0] 1789367083.452994 mapwrite 0x9f1000 thru 0x9fe706 (0xd707 bytes) 000198[0] 1789367083.551450 mapwrite 0x506000 thru 0x5124e1 (0xc4e2 bytes) 000199[1] 1789367083.707490 mapwrite 0x72a000 thru 0x7344a0 (0xa4a1 bytes) 000200[1] 1789367083.870403 mapread 0x660000 thru 0x660bfc (0xbfd bytes) 000202[0] 1789367083.952426 mapread 0x633000 thru 0x63f54b (0xc54c bytes) 000203[1] 1789367084.148103 mapwrite 0x573000 thru 0x57bd6b (0x8d6c bytes) 000204[0] 1789367084.253448 read 0x968000 thru 0x975ca0 (0xdca1 bytes) 000205[0] 1789367084.286050 read 0x902000 thru 0x90f9c5 (0xd9c6 bytes) 000207[0] 1789367084.298405 punch from 0x26ce09 to 0x27a93f (0xdb37 bytes) 000208[1] 1789367084.326055 mapwrite 0x104000 thru 0x106c55 (0x2c56 bytes) 000209[1] 1789367084.431277 write_OD 0x3f9000 thru 0x3fcfff (0x4000 bytes) 000211[0] 1789367084.489008 mapread 0x0b6000 thru 0x0ba053 (0x4054 bytes) 000212[1] 1789367084.518933 punch from 0x4709dd to 0x4736a9 (0x2ccd bytes) 000215[1] 1789367084.593246 mapread 0x5d2000 thru 0x5db465 (0x9466 bytes) 000217[0] 1789367084.615602 write 0x2c9000 thru 0x2d3751 (0xa752 bytes) 000218[1] 1789367084.637221 mapwrite 0x977000 thru 0x9801ca (0x91cb bytes) 000219[0] 1789367084.741248 read 0x3b9000 thru 0x3bdb7d (0x4b7e bytes) 000220[1] 1789367084.765886 mapwrite 0x6c9000 thru 0x6cdb2c (0x4b2d bytes) 000221[1] 1789367084.864598 punch from 0x019d3b to 0x0208c0 (0x6b86 bytes) 000225[0] 1789367084.903882 punch from 0x297526 to 0x29bc28 (0x4703 bytes) 000228[1] 1789367084.938299 mapwrite 0x1e9000 thru 0x1eb07d (0x207e bytes) 000229[0] 1789367085.037037 write 0x749000 thru 0x75410c (0xb10d bytes) 000233[0] 1789367085.099009 punch from 0x0a4e7e to 0x0b1949 (0xcacc bytes) 000234[0] 1789367085.103767 mapread 0x86a000 thru 0x8733d9 (0x93da bytes) 000235[1] 1789367085.154331 mapread 0x6e7000 thru 0x6e7e73 (0xe74 bytes) 000236[0] 1789367085.210053 punch from 0x871b4a to 0x876b3f (0x4ff6 bytes) 000237[0] 1789367085.257648 trunc from 0x9fe707 to 0x7e77c5 (0x216f41 bytes) 000238[0] 1789367085.297725 trunc from 0x7e77c6 to 0x636767 (0x1b105e bytes) 000240[0] 1789367085.323552 mapwrite 0x301000 thru 0x309ec6 (0x8ec7 bytes) 000243[1] 1789367085.407498 punch from 0x0b4fd0 to 0x0bd04d (0x807e bytes) 000245[1] 1789367085.451658 write_OD 0x017000 thru 0x021fff (0xb000 bytes) 000246[1] 1789367085.502042 write_OD 0x73a000 thru 0x73efff (0x5000 bytes) 000250[0] 1789367085.567622 mapwrite 0x686000 thru 0x68d844 (0x7845 bytes) 000251[1] 1789367085.646884 mapread 0x495000 thru 0x497afc (0x2afd bytes) 000252[1] 1789367085.694473 mapwrite 0x02d000 thru 0x03a165 (0xd166 bytes) 000255[0] 1789367085.807434 mapread 0x673000 thru 0x67c339 (0x933a bytes) 000256[0] 1789367085.839965 write 0x89b000 thru 0x8a6e01 (0xbe02 bytes) 000257[0] 1789367085.853251 read 0x35e000 thru 0x35faa6 (0x1aa7 bytes) 000259[1] 1789367085.873730 punch from 0x608ea1 to 0x60e728 (0x5888 bytes) 000260[1] 1789367085.912670 read_OD 0x042000 thru 0x04cfff (0xb000 bytes) 000262[1] 1789367085.939428 mapwrite 0x177000 thru 0x17a6c4 (0x36c5 bytes) 000267[0] 1789367086.074404 write 0x7a9000 thru 0x7b10ce (0x80cf bytes) 000269[1] 1789367086.111142 punch from 0x5a19a8 to 0x5afe82 (0xe4db bytes) 000270[1] 1789367086.134676 read_OD 0x1af000 thru 0x1b1fff (0x3000 bytes) 000271[1] 1789367086.152656 punch from 0x858007 to 0x863ebf (0xbeb9 bytes) 000272[1] 1789367086.169900 read_OD 0x523000 thru 0x52dfff (0xb000 bytes) 000273[0] 1789367086.202074 write 0x685000 thru 0x690760 (0xb761 bytes) 000274[1] 1789367086.227835 mapread 0x61b000 thru 0x622e94 (0x7e95 bytes) 000275[0] 1789367086.316134 write 0x295000 thru 0x2a12e8 (0xc2e9 bytes) 000276[0] 1789367086.341929 mapread 0x769000 thru 0x776ddb (0xdddc bytes) 000277[1] 1789367086.383549 trunc from 0x8a6e02 to 0x5c2826 (0x2e45db bytes) 000278[1] 1789367086.461507 mapread 0x066000 thru 0x0684be (0x24bf bytes) 000280[0] 1789367086.521370 punch from 0x1c2644 to 0x1ca187 (0x7b44 bytes) 000281[1] 1789367086.541747 trunc from 0x5c2827 to 0x11a35c (0x4a84ca bytes) 000283[1] 1789367086.603252 mapwrite 0x789000 thru 0x79716e (0xe16f bytes) 000286[0] 1789367086.900579 trunc from 0x79716f to 0x03bb05 (0x75b669 bytes) 000293[1] 1789367087.008196 punch from 0x032cd7 to 0x03bb05 (0x8e2f bytes) 000295[1] 1789367087.071502 punch from 0x028aec to 0x030f17 (0x842c bytes) 000297[0] 1789367087.119405 trunc from 0x03bb06 to 0x6b9c0e (0x67e109 bytes) 000298[0] 1789367087.210466 mapwrite 0x3c2000 thru 0x3cafd4 (0x8fd5 bytes) 000303[0] 1789367087.498674 write 0x6dd000 thru 0x6def8a (0x1f8b bytes) 000304[0] 1789367087.504575 read 0x235000 thru 0x23c93b (0x793c bytes) 000305[1] 1789367087.547248 mapwrite 0x345000 thru 0x3489bc (0x39bd bytes) 000306[1] 1789367087.776459 trunc from 0x6def8b to 0x867ddd (0x188e53 bytes) 000307[0] 1789367087.812745 mapwrite 0x9b8000 thru 0x9b96af (0x16b0 bytes) 000308[1] 1789367087.914459 mapread 0x40f000 thru 0x413bfc (0x4bfd bytes) 000314[0] 1789367087.987611 read 0x10e000 thru 0x10e519 (0x51a bytes) 000315[1] 1789367088.001880 read_OD 0x907000 thru 0x907fff (0x1000 bytes) 000316[0] 1789367088.008175 mapwrite 0x5de000 thru 0x5edd61 (0xfd62 bytes) 000320[0] 1789367088.275958 write 0x095000 thru 0x0950be (0xbf bytes) 000322[0] 1789367088.401435 punch from 0x9051c3 to 0x9106e6 (0xb524 bytes) 000325[0] 1789367088.445393 mapread 0x7a2000 thru 0x7b0bed (0xebee bytes) 000329[1] 1789367088.539477 write_OD 0x2b3000 thru 0x2b4fff (0x2000 bytes) 000330[1] 1789367088.611142 punch from 0x07b72d to 0x0818cb (0x619f bytes) 000333[0] 1789367088.644485 read 0x1f3000 thru 0x1ff04e (0xc04f bytes) 000335[0] 1789367088.664972 read 0x7d1000 thru 0x7d9a17 (0x8a18 bytes) 000337[1] 1789367088.683161 mapwrite 0x180000 thru 0x186b17 (0x6b18 bytes) 000340[1] 1789367088.891892 read_OD 0x3ef000 thru 0x3f8fff (0xa000 bytes) 000342[0] 1789367088.932866 mapwrite 0x883000 thru 0x88c747 (0x9748 bytes) 000343[1] 1789367089.065335 trunc from 0x9b96b0 to 0x8ed047 (0xcc668 bytes) 000344[0] 1789367089.099917 mapread 0x699000 thru 0x6a6c1e (0xdc1f bytes) 000345[1] 1789367089.154063 read_OD 0x320000 thru 0x329fff (0xa000 bytes) 000346[1] 1789367089.168822 trunc from 0x8ed048 to 0x7d7cd8 (0x11536f bytes) 000348[0] 1789367089.216557 write 0x352000 thru 0x35431b (0x231c bytes) 000349[0] 1789367089.253357 write 0x1cb000 thru 0x1d1f13 (0x6f14 bytes) 000350[0] 1789367089.275493 read 0x58d000 thru 0x58fc9d (0x2c9e bytes) 000351[0] 1789367089.288522 punch from 0x3ff807 to 0x40dfd0 (0xe7ca bytes) 000352[0] 1789367089.321189 write 0x40a000 thru 0x410992 (0x6993 bytes) 000353[0] 1789367089.328393 mapread 0x271000 thru 0x280d67 (0xfd68 bytes) 000355[0] 1789367089.363700 punch from 0x702834 to 0x705d3f (0x350c bytes) 000358[1] 1789367089.432209 punch from 0x32b0d7 to 0x333f3e (0x8e68 bytes) 000359[0] 1789367089.456613 mapwrite 0x268000 thru 0x2750d9 (0xd0da bytes) 000360[1] 1789367089.591587 write_OD 0x613000 thru 0x61afff (0x8000 bytes) 000361[1] 1789367089.720634 write_OD 0x77a000 thru 0x784fff (0xb000 bytes) 000364[0] 1789367089.763942 write 0x423000 thru 0x429bea (0x6beb bytes) 000365[0] 1789367089.782371 mapwrite 0x60c000 thru 0x61ad39 (0xed3a bytes) 000366[1] 1789367089.848888 punch from 0x7c6593 to 0x7c6a7d (0x4eb bytes) 000367[0] 1789367089.896171 write 0x855000 thru 0x85574f (0x750 bytes) 000368[1] 1789367089.915592 trunc from 0x855750 to 0x38a15b (0x4cb5f4 bytes) 000371[0] 1789367090.012984 punch from 0x241380 to 0x24c301 (0xaf82 bytes) 000372[1] 1789367090.045105 mapwrite 0x5a7000 thru 0x5b6511 (0xf512 bytes) 000374[0] 1789367090.280598 mapwrite 0x19e000 thru 0x1add5d (0xfd5e bytes) 000376[1] 1789367090.400114 mapwrite 0x482000 thru 0x48e16d (0xc16e bytes) 000378[0] 1789367090.532585 write 0x4f4000 thru 0x501886 (0xd887 bytes) 000379[1] 1789367090.567171 trunc from 0x5b6512 to 0x60b2c5 (0x54db4 bytes) 000380[0] 1789367090.626200 trunc from 0x60b2c6 to 0x0c84e7 (0x542dde bytes) 000382[0] 1789367090.678779 write 0x904000 thru 0x90dab7 (0x9ab8 bytes) 000384[0] 1789367090.721389 write 0x18b000 thru 0x1974d6 (0xc4d7 bytes) 000385[0] 1789367090.741636 mapwrite 0x481000 thru 0x48389f (0x28a0 bytes) 000386[0] 1789367090.821497 write 0x5c3000 thru 0x5cf6e4 (0xc6e5 bytes) 000387[0] 1789367090.847915 punch from 0x08de7e to 0x09a3c9 (0xc54c bytes) 000388[0] 1789367090.853169 write 0x416000 thru 0x423af9 (0xdafa bytes) 000389[0] 1789367090.876185 mapread 0x512000 thru 0x518b6d (0x6b6e bytes) 000391[1] 1789367090.903949 read_OD 0x712000 thru 0x71efff (0xd000 bytes) 000394[1] 1789367090.951192 punch from 0x443df3 to 0x452b74 (0xed82 bytes) 000395[0] 1789367090.980007 mapread 0x3b9000 thru 0x3bd1ce (0x41cf bytes) 000396[1] 1789367091.016569 read_OD 0x4d8000 thru 0x4dcfff (0x5000 bytes) 000397[0] 1789367091.026911 mapwrite 0x006000 thru 0x00ffc9 (0x9fca bytes) 000398[1] 1789367091.129745 write_OD 0x7a5000 thru 0x7b3fff (0xf000 bytes) 000399[0] 1789367091.192141 write 0x2a9000 thru 0x2b4055 (0xb056 bytes) 000401[1] 1789367091.212865 mapread 0x3d7000 thru 0x3d96b8 (0x26b9 bytes) 000402[1] 1789367091.263776 punch from 0x4ac82f to 0x4b63d3 (0x9ba5 bytes) 000404[1] 1789367091.285360 punch from 0x5de91e to 0x5ea600 (0xbce3 bytes) 000405[1] 1789367091.290543 punch from 0x732a45 to 0x738f36 (0x64f2 bytes) 000408[0] 1789367091.325575 punch from 0x6ed218 to 0x6f6784 (0x956d bytes) 000409[1] 1789367091.371320 punch from 0x7b821e to 0x7c4f6a (0xcd4d bytes) 000411[0] 1789367091.417961 trunc from 0x90dab8 to 0x81ba6f (0xf2048 bytes) 000413[0] 1789367091.468749 mapread 0x14e000 thru 0x14e97c (0x97d bytes) 000414[0] 1789367091.474570 mapwrite 0x6f9000 thru 0x6fea4a (0x5a4b bytes) 000415[0] 1789367091.545856 write 0x765000 thru 0x76d5e9 (0x85ea bytes) 000416[1] 1789367091.551998 write_OD 0x841000 thru 0x846fff (0x6000 bytes) 000418[0] 1789367091.649155 read 0x13b000 thru 0x146d92 (0xbd93 bytes) 000422[1] 1789367091.663406 mapwrite 0x87f000 thru 0x886919 (0x791a bytes) 000423[1] 1789367091.808408 mapwrite 0x7c4000 thru 0x7c6322 (0x2323 bytes) 000424[1] 1789367091.977310 write_OD 0x6d5000 thru 0x6dcfff (0x8000 bytes) 000425[1] 1789367092.073580 write_OD 0x7d4000 thru 0x7dbfff (0x8000 bytes) 000432[0] 1789367092.140197 mapread 0x106000 thru 0x115092 (0xf093 bytes) 000433[1] 1789367092.186580 write_OD 0x95f000 thru 0x965fff (0x7000 bytes) 000434[1] 1789367092.267606 trunc from 0x966000 to 0x00ca7f (0x959580 bytes) 000437[1] 1789367092.434360 write_OD 0x514000 thru 0x51afff (0x7000 bytes) 000438[0] 1789367092.593715 mapwrite 0x82d000 thru 0x833081 (0x6082 bytes) 000440[1] 1789367092.846392 trunc from 0x833082 to 0x615d21 (0x21d360 bytes) 000441[1] 1789367092.967324 mapread 0x1fc000 thru 0x207c6b (0xbc6c bytes) 000442[1] 1789367093.063539 trunc from 0x615d22 to 0x84582c (0x22fb0b bytes) 000444[1] 1789367093.090414 mapread 0x00b000 thru 0x00c916 (0x1917 bytes) 000445[1] 1789367093.102464 trunc from 0x84582d to 0x7deec1 (0x6696b bytes) 000446[1] 1789367093.140021 read_OD 0x2b8000 thru 0x2bbfff (0x4000 bytes) 000447[1] 1789367093.170280 read_OD 0x0cc000 thru 0x0d1fff (0x6000 bytes) 000448[0] 1789367093.187366 mapread 0x46c000 thru 0x478a9a (0xca9b bytes) 000449[1] 1789367093.218967 punch from 0x336cd4 to 0x33b042 (0x436f bytes) 000450[0] 1789367093.243767 punch from 0x0b9830 to 0x0be6a4 (0x4e75 bytes) 000451[1] 1789367093.264241 mapread 0x7a5000 thru 0x7a5e88 (0xe89 bytes) 000454[1] 1789367093.295539 mapread 0x111000 thru 0x11a8d2 (0x98d3 bytes) 000457[1] 1789367093.317029 trunc from 0x7deec2 to 0x2221db (0x5bcce6 bytes) 000458[1] 1789367093.354442 write_OD 0x765000 thru 0x771fff (0xd000 bytes) 000464[1] 1789367093.412875 read_OD 0x153000 thru 0x15afff (0x8000 bytes) 000467[1] 1789367093.439881 read_OD 0x031000 thru 0x031fff (0x1000 bytes) 000469[0] 1789367093.446875 mapread 0x0ca000 thru 0x0d360b (0x960c bytes) 000470[1] 1789367093.470131 punch from 0x721f7f to 0x728e30 (0x6eb2 bytes) 000472[0] 1789367093.511912 punch from 0x474357 to 0x47a5bf (0x6269 bytes) 000474[0] 1789367093.564356 trunc from 0x772000 to 0x99da3a (0x22ba3b bytes) 000476[1] 1789367093.637257 punch from 0x504704 to 0x50e003 (0x9900 bytes) 000482[0] 1789367093.806465 mapread 0x4ae000 thru 0x4b53e9 (0x73ea bytes) 000484[0] 1789367093.908742 punch from 0x554d9d to 0x559ccc (0x4f30 bytes) 000485[1] 1789367093.964760 punch from 0x7df323 to 0x7ec6d5 (0xd3b3 bytes) 000486[1] 1789367094.031017 read_OD 0x483000 thru 0x484fff (0x2000 bytes) 000487[1] 1789367094.072211 mapread 0x36e000 thru 0x37217b (0x417c bytes) 000488[1] 1789367094.118444 write_OD 0x4b7000 thru 0x4b7fff (0x1000 bytes) 000489[1] 1789367094.216265 trunc from 0x99da3b to 0x5dceac (0x3c0b8e bytes) 000491[0] 1789367094.319906 write 0x436000 thru 0x43b7c4 (0x57c5 bytes) 000493[1] 1789367094.424457 mapread 0x566000 thru 0x574751 (0xe752 bytes) 000494[1] 1789367094.531105 read_OD 0x3c8000 thru 0x3d0fff (0x9000 bytes) 000495[0] 1789367094.547628 trunc from 0x5dcead to 0x36cef0 (0x26ffbc bytes) 000497[1] 1789367094.644309 write_OD 0x799000 thru 0x7a2fff (0xa000 bytes) 000498[1] 1789367094.765879 read_OD 0x446000 thru 0x446fff (0x1000 bytes) 000500[1] 1789367094.783864 mapread 0x24c000 thru 0x25373c (0x773d bytes) All operations completed A-OK! PASS 16a (54s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:25:00 (1789367100) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.361985 s, 23.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.389745 s, 21.5 MB/s Chance of close/open is 1 in 50 Seed set to 7104 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x6ee6db 000002[1] 1789367104.914626 trunc from 00000000 to 0x6ee6da (0x6ee6db bytes) 000004[1] 1789367104.943577 trunc from 0x6ee6db to 0x4cf929 (0x21edb1 bytes) 000006[0] 1789367104.963293 mapwrite 0x378531 thru 0x3798c1 (0x1391 bytes) 000008[0] 1789367105.149449 read 0x2637cd thru 0x2643b8 (0xbec bytes) 000009[1] 1789367105.167140 write 0x1e1d1c thru 0x1e2877 (0xb5c bytes) 000011[1] 1789367105.221269 trunc from 0x4cf92a to 0x633ed8 (0x1645af bytes) 000013[0] 1789367105.241287 mapwrite 0x6f3640 thru 0x6f5267 (0x1c28 bytes) 000014[0] 1789367105.408411 punch from 0x2f005f to 0x2f1854 (0x17f6 bytes) 000015[0] 1789367105.417278 write 0x88aaad thru 0x88bb5b (0x10af bytes) truncating to largest ever: 0x84d541 000017[1] 1789367105.455053 trunc from 0x88bb5c to 0x84d540 (0x3e61b bytes) 000018[1] 1789367105.550435 write 0x357382 thru 0x357d2a (0x9a9 bytes) 000019[0] 1789367105.562479 mapread 0x604f9c thru 0x6060be (0x1123 bytes) 000021[0] 1789367105.664414 trunc from 0x84d541 to 0x82f9b5 (0x1db8b bytes) 000022[0] 1789367105.774562 write 0x95c01d thru 0x95c5c7 (0x5ab bytes) 000023[0] 1789367105.813885 mapread 0x4d7e7d thru 0x4d9a41 (0x1bc5 bytes) 000025[1] 1789367105.905636 read 0x2f8c88 thru 0x2f9118 (0x491 bytes) 000027[0] 1789367105.958628 read 0x6794df thru 0x67b0d6 (0x1bf8 bytes) 000029[1] 1789367105.982321 punch from 0x03e1fd to 0x03ff67 (0x1d6b bytes) 000037[1] 1789367106.180092 write 0x7bdcf1 thru 0x7be8e1 (0xbf1 bytes) 000039[1] 1789367106.237549 read 0x62d9fe thru 0x62eb89 (0x118c bytes) 000041[1] 1789367106.245749 read 0x033b33 thru 0x03411a (0x5e8 bytes) truncating to largest ever: 0x890678 000043[1] 1789367106.277523 trunc from 0x95c5c8 to 0x890677 (0xcbf50 bytes) 000044[1] 1789367106.295678 read 0x2fb0f1 thru 0x2fc5cc (0x14dc bytes) 000045[1] 1789367106.332171 punch from 0x4b0309 to 0x4b10ed (0xde5 bytes) 000046[1] 1789367106.339317 read 0x497efc thru 0x498049 (0x14e bytes) 000047[1] 1789367106.371984 write 0x480ce2 thru 0x481fb5 (0x12d4 bytes) 000048[1] 1789367106.382883 punch from 0x276131 to 0x276bb5 (0xa85 bytes) 000049[1] 1789367106.387445 read 0x297068 thru 0x298147 (0x10e0 bytes) 000050[0] 1789367106.394278 mapread 0x6a9e86 thru 0x6ab042 (0x11bd bytes) 000053[0] 1789367106.465154 write 0x389805 thru 0x389be9 (0x3e5 bytes) 000055[0] 1789367106.503603 trunc from 0x890678 to 0x6cd800 (0x1c2e77 bytes) truncating to largest ever: 0x8a5978 000056[1] 1789367106.557122 trunc from 0x6cd801 to 0x8a5977 (0x1d8177 bytes) 000057[0] 1789367106.603239 read 0x35ef50 thru 0x35fba2 (0xc53 bytes) 000058[1] 1789367106.661699 mapwrite 0x14844d thru 0x14a291 (0x1e45 bytes) 000061[0] 1789367106.824983 punch from 0x82092e to 0x822889 (0x1f5c bytes) 000062[1] 1789367106.868509 write 0x541071 thru 0x54141c (0x3ac bytes) 000063[1] 1789367106.902571 punch from 0x12156e to 0x1227fd (0x1290 bytes) 000065[1] 1789367106.959130 trunc from 0x8a5978 to 0x5ba9ee (0x2eaf89 bytes) 000067[1] 1789367106.982076 punch from 0x010bec to 0x011b8f (0xfa4 bytes) 000068[1] 1789367106.989749 read 0x4cdc7f thru 0x4cdeb6 (0x238 bytes) 000069[0] 1789367107.018117 punch from 0x1754ea to 0x17646a (0xf81 bytes) 000071[0] 1789367107.077533 punch from 0x221c8d to 0x223aab (0x1e1f bytes) 000073[0] 1789367107.137243 mapread 0x3d8faa thru 0x3da536 (0x158d bytes) 000074[1] 1789367107.187920 punch from 0x1943a6 to 0x195069 (0xcc4 bytes) 000076[1] 1789367107.242593 read 0x3f1185 thru 0x3f278d (0x1609 bytes) 000078[0] 1789367107.250229 mapread 0x38e7ef thru 0x39070c (0x1f1e bytes) 000079[1] 1789367107.288485 read 0x2f9d64 thru 0x2fab87 (0xe24 bytes) 000081[1] 1789367107.311434 mapwrite 0x632ecc thru 0x6346f4 (0x1829 bytes) 000083[1] 1789367107.474908 read 0x320eca thru 0x3226b3 (0x17ea bytes) 000085[0] 1789367107.520743 mapwrite 0x2b1f11 thru 0x2b3ca1 (0x1d91 bytes) 000086[0] 1789367107.686035 read 0x54365e thru 0x544bb7 (0x155a bytes) 000088[0] 1789367107.707588 mapread 0x56dbfd thru 0x56eb17 (0xf1b bytes) 000091[1] 1789367107.735893 mapwrite 0x9c638c thru 0x9c6586 (0x1fb bytes) 000093[1] 1789367107.913658 write 0x78f77b thru 0x79035b (0xbe1 bytes) 000094[1] 1789367107.946572 mapwrite 0x417e33 thru 0x419298 (0x1466 bytes) 000095[0] 1789367108.064837 mapread 0x9adea9 thru 0x9aeb96 (0xcee bytes) 000096[1] 1789367108.112921 read 0x4633bf thru 0x463719 (0x35b bytes) 000097[0] 1789367108.149845 read 0x99800e thru 0x999a83 (0x1a76 bytes) 000100[1] 1789367108.186632 trunc from 0x9c6587 to 0x75d1e3 (0x2693a3 bytes) 000101[0] 1789367108.263383 mapread 0x410afe thru 0x411d73 (0x1276 bytes) 000102[1] 1789367108.319290 trunc from 0x75d1e4 to 0x46cdc3 (0x2f0420 bytes) 000105[0] 1789367108.382206 write 0x68d573 thru 0x68d6bc (0x14a bytes) 000106[0] 1789367108.414593 mapwrite 0x313876 thru 0x313f39 (0x6c4 bytes) 000107[1] 1789367108.496468 punch from 0x636a3c to 0x637b4c (0x1111 bytes) 000109[1] 1789367108.553455 read 0x154043 thru 0x15460a (0x5c8 bytes) 000110[1] 1789367108.571270 mapwrite 0x3aeb9d thru 0x3af5a0 (0xa04 bytes) 000111[1] 1789367108.672449 read 0x141cbb thru 0x1437a2 (0x1ae8 bytes) 000115[1] 1789367108.720319 read 0x1845a6 thru 0x1852c9 (0xd24 bytes) 000117[1] 1789367108.777189 mapread 0x2db94a thru 0x2dc162 (0x819 bytes) 000118[0] 1789367108.790141 trunc from 0x68d6bd to 0x536d56 (0x156966 bytes) 000119[0] 1789367108.866251 write 0x32bd1d thru 0x32d0e2 (0x13c6 bytes) 000120[1] 1789367108.904250 mapread 0x2e0ebc thru 0x2e2bf5 (0x1d3a bytes) 000121[1] 1789367108.971109 mapread 0x486008 thru 0x4876b9 (0x16b2 bytes) 000123[0] 1789367109.002349 read 0x170de3 thru 0x170f29 (0x147 bytes) truncating to largest ever: 0x9e8c65 000124[0] 1789367109.008012 trunc from 0x536d57 to 0x9e8c64 (0x4b1f0e bytes) 000126[0] 1789367109.092489 punch from 0x754603 to 0x755ce6 (0x16e4 bytes) 000128[1] 1789367109.118928 write 0x530e45 thru 0x532b27 (0x1ce3 bytes) 000129[1] 1789367109.162442 trunc from 0x9e8c65 to 0x960a9d (0x881c7 bytes) 000130[1] 1789367109.214633 read 0x834bf2 thru 0x8353aa (0x7b9 bytes) 000131[1] 1789367109.275720 punch from 0x37b8e1 to 0x37bf4d (0x66d bytes) 000132[1] 1789367109.317230 write 0x671e15 thru 0x673ab9 (0x1ca5 bytes) 000133[0] 1789367109.329099 mapread 0x3ed4e4 thru 0x3ee0f0 (0xc0d bytes) 000134[0] 1789367109.378473 trunc from 0x960a9e to 0x540add (0x41ffc0 bytes) 000137[0] 1789367109.450030 punch from 0x29287c to 0x294205 (0x198a bytes) 000138[1] 1789367109.479195 punch from 0x296ca1 to 0x298124 (0x1484 bytes) 000140[0] 1789367109.516603 trunc from 0x540ade to 0x79305f (0x252582 bytes) 000142[0] 1789367109.591471 punch from 0x123bd1 to 0x124610 (0xa40 bytes) 000143[1] 1789367109.604308 punch from 0x52867e to 0x528be5 (0x568 bytes) 000145[1] 1789367109.679049 mapread 0x0ce45f thru 0x0cfb61 (0x1703 bytes) 000146[0] 1789367109.736381 trunc from 0x793060 to 0x8a2175 (0x10f116 bytes) 000147[1] 1789367109.821268 write 0x423c74 thru 0x424970 (0xcfd bytes) 000148[0] 1789367109.865849 write 0x96d071 thru 0x96d374 (0x304 bytes) 000149[0] 1789367109.881017 write 0x28c5a3 thru 0x28dc45 (0x16a3 bytes) 000151[1] 1789367109.932546 mapwrite 0x54ff14 thru 0x5502fd (0x3ea bytes) 000152[1] 1789367110.016386 trunc from 0x96d375 to 0x843433 (0x129f41 bytes) 000153[1] 1789367110.116498 mapwrite 0x7e6cd1 thru 0x7e8799 (0x1ac9 bytes) 000155[1] 1789367110.232605 mapwrite 0x7a74ce thru 0x7a7d90 (0x8c3 bytes) 000157[1] 1789367110.297607 punch from 0x2ccdf7 to 0x2cde18 (0x1022 bytes) 000158[1] 1789367110.302520 read 0x62ac0f thru 0x62ca68 (0x1e5a bytes) 000160[0] 1789367110.330454 mapwrite 0x67aff8 thru 0x67c231 (0x123a bytes) 000161[1] 1789367110.419008 mapread 0x6f4b31 thru 0x6f53db (0x8ab bytes) 000163[0] 1789367110.496718 mapread 0x4016ab thru 0x403429 (0x1d7f bytes) 000164[0] 1789367110.551655 trunc from 0x843434 to 0x650c16 (0x1f281d bytes) 000165[0] 1789367110.665692 write 0x4193f0 thru 0x41993f (0x550 bytes) 000167[0] 1789367110.721764 mapread 0x06b2e5 thru 0x06cab3 (0x17cf bytes) 000168[0] 1789367110.752134 punch from 0x30513e to 0x305329 (0x1ec bytes) 000169[1] 1789367110.780093 write 0x7b42cf thru 0x7b4b27 (0x859 bytes) 000177[1] 1789367110.998718 trunc from 0x7b4b28 to 0x5f4c88 (0x1bfe9f bytes) 000178[1] 1789367111.055954 write 0x3d1933 thru 0x3d2e6c (0x153a bytes) 000180[0] 1789367111.128527 punch from 0x568ad4 to 0x569f30 (0x145d bytes) 000182[0] 1789367111.178399 mapread 0x07ed04 thru 0x07f5d7 (0x8d4 bytes) 000183[1] 1789367111.248349 mapread 0x1e1db5 thru 0x1e21cb (0x417 bytes) 000185[0] 1789367111.290523 mapwrite 0x8962fd thru 0x8976ff (0x1403 bytes) 000186[0] 1789367111.389542 mapwrite 0x7a3371 thru 0x7a385c (0x4ec bytes) 000190[1] 1789367111.471272 trunc from 0x897700 to 0x184cc8 (0x712a37 bytes) 000191[0] 1789367111.520983 read 0x0a61b1 thru 0x0a7f5f (0x1daf bytes) 000193[1] 1789367111.543531 write 0x7931fa thru 0x793a96 (0x89d bytes) 000195[0] 1789367111.581329 mapread 0x2d95c4 thru 0x2db057 (0x1a94 bytes) 000196[1] 1789367111.645585 write 0x6b92d3 thru 0x6b9b69 (0x897 bytes) 000198[0] 1789367111.685982 read 0x308dbe thru 0x30a5e8 (0x182b bytes) 000199[0] 1789367111.706863 punch from 0x0a2bf3 to 0x0a3a0d (0xe1b bytes) 000200[1] 1789367111.739746 mapwrite 0x5ce035 thru 0x5ce745 (0x711 bytes) 000201[0] 1789367111.809591 punch from 0x2abba9 to 0x2ac945 (0xd9d bytes) 000202[1] 1789367111.836917 punch from 0x6253fc to 0x626433 (0x1038 bytes) 000203[0] 1789367111.865380 write 0x4e1fbf thru 0x4e385f (0x18a1 bytes) 000204[1] 1789367111.927401 punch from 0x1b289d to 0x1b30cf (0x833 bytes) 000205[1] 1789367111.975383 mapread 0x093eb1 thru 0x094ba4 (0xcf4 bytes) 000209[0] 1789367112.094514 read 0x115caf thru 0x116d66 (0x10b8 bytes) 000210[1] 1789367112.148617 punch from 0x2a5dd1 to 0x2a6011 (0x241 bytes) 000211[0] 1789367112.192486 mapread 0x1299a3 thru 0x129f8d (0x5eb bytes) 000212[0] 1789367112.228079 write 0x028a31 thru 0x028b2a (0xfa bytes) 000214[1] 1789367112.312426 trunc from 0x793a97 to 0x1e8f24 (0x5aab72 bytes) 000217[1] 1789367112.493299 trunc from 0x1e8f25 to 0x858fed (0x6700c9 bytes) 000218[1] 1789367112.552971 mapwrite 0x5334dc thru 0x5335fa (0x11f bytes) 000219[0] 1789367112.675742 mapread 0x1daffb thru 0x1dccfa (0x1d00 bytes) 000222[1] 1789367112.848812 read 0x03faf0 thru 0x040951 (0xe62 bytes) 000224[0] 1789367112.907525 punch from 0x834d38 to 0x83508c (0x355 bytes) 000226[1] 1789367113.011153 mapwrite 0x8898b3 thru 0x88af36 (0x1684 bytes) 000228[1] 1789367113.156735 trunc from 0x88af37 to 0x98269b (0xf7765 bytes) 000229[1] 1789367113.206075 trunc from 0x98269c to 0x3c5629 (0x5bd072 bytes) 000230[1] 1789367113.231076 mapwrite 0x16e5b0 thru 0x16eba3 (0x5f4 bytes) 000231[1] 1789367113.313138 trunc from 0x3c562a to 0x19de09 (0x227820 bytes) 000233[1] 1789367113.343308 write 0x2cc447 thru 0x2ce1f3 (0x1dad bytes) 000234[0] 1789367113.345022 trunc from 0x2ce1f4 to 0x964914 (0x696721 bytes) 000236[1] 1789367113.519230 trunc from 0x964915 to 0x2b47dc (0x6b0138 bytes) 000237[0] 1789367113.646271 write 0x4e94ef thru 0x4ea060 (0xb72 bytes) 000238[1] 1789367113.742534 mapread 0x348d75 thru 0x3495ea (0x876 bytes) 000239[1] 1789367113.871492 trunc from 0x4ea061 to 0x596712 (0xac6b2 bytes) 000240[0] 1789367113.923968 mapread 0x0fcb87 thru 0x0fd019 (0x493 bytes) 000241[1] 1789367113.979724 mapread 0x102d91 thru 0x1042e3 (0x1553 bytes) 000244[1] 1789367114.030635 mapread 0x0af4bd thru 0x0b084f (0x1393 bytes) 000245[1] 1789367114.042513 mapwrite 0x40b672 thru 0x40c1d7 (0xb66 bytes) 000246[1] 1789367114.109074 write 0x375be4 thru 0x37632b (0x748 bytes) 000248[0] 1789367114.157501 punch from 0x05e7e7 to 0x05f05c (0x876 bytes) 000250[1] 1789367114.214577 punch from 0x06be24 to 0x06d7c8 (0x19a5 bytes) 000252[0] 1789367114.232216 punch from 0x463f9c to 0x464e7b (0xee0 bytes) 000255[1] 1789367114.300312 trunc from 0x596713 to 0x22ec70 (0x367aa2 bytes) 000261[0] 1789367114.424134 mapread 0x1c58aa thru 0x1c7756 (0x1ead bytes) 000266[0] 1789367114.513205 trunc from 0x22ec71 to 0x4e36ea (0x2b4a7a bytes) 000268[1] 1789367114.614657 punch from 0x3ec534 to 0x3ed17a (0xc47 bytes) 000270[0] 1789367114.668329 mapwrite 0x401483 thru 0x4021d2 (0xd50 bytes) 000272[0] 1789367114.754229 mapread 0x2addd5 thru 0x2ae4d8 (0x704 bytes) 000274[0] 1789367114.800487 read 0x1c45d4 thru 0x1c5309 (0xd36 bytes) 000276[0] 1789367114.839123 trunc from 0x4e36eb to 0x18825c (0x35b48e bytes) 000277[0] 1789367114.893662 read 0x08a8a2 thru 0x08bbca (0x1329 bytes) 000279[0] 1789367115.040517 read 0x0a2823 thru 0x0a40f4 (0x18d2 bytes) 000281[1] 1789367115.110182 mapwrite 0x27e94c thru 0x27edf6 (0x4ab bytes) 000283[1] 1789367115.264750 punch from 0x15340d to 0x1535ac (0x1a0 bytes) 000285[1] 1789367115.302187 trunc from 0x27edf7 to 0x094777 (0x1ea67f bytes) 000286[1] 1789367115.349666 read 0x033b7c thru 0x03486c (0xcf1 bytes) 000288[0] 1789367115.392934 mapwrite 0x8837dc thru 0x8850cf (0x18f4 bytes) 000291[0] 1789367115.674211 write 0x9fac34 thru 0x9fb92a (0xcf7 bytes) 000292[0] 1789367115.705293 read 0x44dad0 thru 0x44fa5f (0x1f90 bytes) 000293[0] 1789367115.705909 mapwrite 0x939e8e thru 0x93b709 (0x187c bytes) 000296[0] 1789367115.815371 mapwrite 0x652eaa thru 0x652ec1 (0x18 bytes) 000299[0] 1789367115.970666 read 0x7eae3b thru 0x7ec2e4 (0x14aa bytes) 000301[0] 1789367115.971653 read 0x640db3 thru 0x64236f (0x15bd bytes) 000302[0] 1789367115.972113 write 0x15e398 thru 0x160184 (0x1ded bytes) 000303[1] 1789367116.000673 mapread 0x5bcde9 thru 0x5bd78d (0x9a5 bytes) 000304[0] 1789367116.048594 mapread 0x91d1d7 thru 0x91df43 (0xd6d bytes) 000306[1] 1789367116.054659 write 0x3d1b3f thru 0x3d2e0f (0x12d1 bytes) 000307[1] 1789367116.168208 mapwrite 0x0fd6e0 thru 0x0fefc5 (0x18e6 bytes) 000308[0] 1789367116.256493 write 0x0119b0 thru 0x011b18 (0x169 bytes) 000309[0] 1789367116.315333 trunc from 0x9fb92b to 0x6be589 (0x33d3a1 bytes) 000310[0] 1789367116.368624 mapwrite 0x537096 thru 0x5386e5 (0x1650 bytes) 000313[1] 1789367116.521312 read 0x527037 thru 0x5283de (0x13a8 bytes) 000316[0] 1789367116.611466 mapwrite 0x50c554 thru 0x50d27b (0xd28 bytes) 000318[1] 1789367116.724514 write 0x83c9db thru 0x83cc6b (0x291 bytes) 000320[0] 1789367116.819666 mapread 0x81258c thru 0x813c46 (0x16bb bytes) 000322[1] 1789367116.886586 trunc from 0x83cc6c to 0x71d222 (0x11fa49 bytes) 000323[0] 1789367116.970751 punch from 0x0d0822 to 0x0d10c9 (0x8a8 bytes) 000324[1] 1789367117.006770 punch from 0x35724e to 0x358088 (0xe3b bytes) 000326[1] 1789367117.063454 trunc from 0x71d223 to 0x441729 (0x2dbaf9 bytes) 000327[1] 1789367117.104537 mapwrite 0x474a76 thru 0x476a44 (0x1fcf bytes) 000328[1] 1789367117.194686 mapwrite 0x23ef63 thru 0x23fe2b (0xec9 bytes) 000329[1] 1789367117.299425 write 0x74eedf thru 0x750a56 (0x1b78 bytes) 000330[1] 1789367117.307338 write 0x4759c3 thru 0x476143 (0x781 bytes) 000331[1] 1789367117.326500 read 0x06e65c thru 0x06f521 (0xec6 bytes) 000332[0] 1789367117.352271 mapread 0x1f1e85 thru 0x1f1f94 (0x110 bytes) 000334[1] 1789367117.409141 trunc from 0x750a57 to 0x70b607 (0x4544f bytes) 000335[0] 1789367117.454211 mapwrite 0x415426 thru 0x4165c8 (0x11a3 bytes) 000336[0] 1789367117.577157 mapread 0x3758da thru 0x376982 (0x10a9 bytes) 000338[0] 1789367117.606818 punch from 0x31ff36 to 0x320d8c (0xe57 bytes) 000342[1] 1789367117.666387 mapwrite 0x83a90d thru 0x83b572 (0xc66 bytes) 000343[1] 1789367117.807886 mapwrite 0x3ef2e8 thru 0x3f0470 (0x1189 bytes) 000344[1] 1789367117.883314 punch from 0x340833 to 0x340dc2 (0x590 bytes) 000346[1] 1789367117.956343 mapread 0x3475b6 thru 0x34904f (0x1a9a bytes) 000354[0] 1789367118.178446 mapwrite 0x64898b thru 0x649c5d (0x12d3 bytes) 000357[0] 1789367118.318091 punch from 0x100ac6 to 0x102452 (0x198d bytes) 000359[1] 1789367118.359462 write 0x528854 thru 0x52904f (0x7fc bytes) 000362[0] 1789367118.447904 trunc from 0x83b573 to 0x173cca (0x6c78a8 bytes) 000363[0] 1789367118.508446 trunc from 0x173ccb to 0x3cb6e9 (0x257a1f bytes) 000364[1] 1789367118.566047 read 0x14d77f thru 0x14e9e6 (0x1268 bytes) 000365[0] 1789367118.661056 mapread 0x2cae0a thru 0x2cc826 (0x1a1d bytes) 000366[1] 1789367118.684505 mapread 0x0c81a2 thru 0x0c9522 (0x1381 bytes) 000368[1] 1789367118.711169 read 0x3a15e3 thru 0x3a2d6f (0x178d bytes) 000369[1] 1789367118.712026 mapread 0x3450e8 thru 0x346a6e (0x1987 bytes) 000371[1] 1789367118.733223 write 0x56be4a thru 0x56dbc7 (0x1d7e bytes) 000375[0] 1789367118.771986 trunc from 0x56dbc8 to 0x00246a (0x56b75d bytes) 000376[1] 1789367118.818204 read 0x001ed1 thru 0x00246a (0x59a bytes) 000377[0] 1789367118.851311 punch from 0x0021b7 to 0x00246a (0x2b4 bytes) 000378[1] 1789367118.871670 write 0x0e2bfd thru 0x0e3c33 (0x1037 bytes) 000379[0] 1789367118.903938 write 0x2073cd thru 0x208e9e (0x1ad2 bytes) 000380[0] 1789367118.949085 mapread 0x1299d7 thru 0x129f45 (0x56f bytes) 000383[0] 1789367118.977385 mapwrite 0x250846 thru 0x251b5a (0x1315 bytes) 000385[1] 1789367119.094633 trunc from 0x251b5b to 0x65fe4d (0x40e2f3 bytes) 000386[0] 1789367119.141259 mapwrite 0x1bba04 thru 0x1bba56 (0x53 bytes) 000387[1] 1789367119.222373 read 0x4e432c thru 0x4e4f62 (0xc37 bytes) 000390[1] 1789367119.283979 trunc from 0x65fe4e to 0x0518d1 (0x60e57c bytes) 000391[1] 1789367119.330769 mapread 0x04be13 thru 0x04db30 (0x1d1e bytes) 000394[0] 1789367119.409605 punch from 0x0161a4 to 0x01656a (0x3c7 bytes) 000395[0] 1789367119.443100 mapread 0x007b9f thru 0x00871e (0xb80 bytes) 000396[0] 1789367119.477393 write 0x628896 thru 0x629b41 (0x12ac bytes) 000397[1] 1789367119.521237 trunc from 0x629b42 to 0x3fb0ca (0x22ea77 bytes) 000399[1] 1789367119.566708 trunc from 0x3fb0cb to 0x355ae7 (0xa55e3 bytes) 000401[0] 1789367119.624538 punch from 0x0fe6e0 to 0x0fe96f (0x290 bytes) 000402[1] 1789367119.694348 punch from 0x250221 to 0x25164c (0x142c bytes) 000404[0] 1789367119.754621 read 0x04cb48 thru 0x04d697 (0xb50 bytes) 000405[0] 1789367119.901022 punch from 0x351b2c to 0x3528c4 (0xd99 bytes) 000406[1] 1789367119.953538 mapwrite 0x70fcb7 thru 0x71012f (0x479 bytes) 000407[0] 1789367120.085144 read 0x694a1c thru 0x696244 (0x1829 bytes) 000409[0] 1789367120.141265 read 0x5f0651 thru 0x5f06b4 (0x64 bytes) 000410[0] 1789367120.146721 mapwrite 0x964656 thru 0x965239 (0xbe4 bytes) 000411[0] 1789367120.256717 mapwrite 0x333ada thru 0x333b41 (0x68 bytes) 000414[1] 1789367120.335070 write 0x3746f2 thru 0x374bc8 (0x4d7 bytes) 000415[0] 1789367120.368847 punch from 0x6cfdcd to 0x6d1d22 (0x1f56 bytes) 000418[0] 1789367120.431324 read 0x96015a thru 0x96027c (0x123 bytes) 000421[1] 1789367120.455025 mapread 0x40a9ed thru 0x40bfe0 (0x15f4 bytes) 000424[1] 1789367120.483179 read 0x6c261c thru 0x6c2c09 (0x5ee bytes) 000426[0] 1789367120.493225 trunc from 0x96523a to 0x38fbdf (0x5d565a bytes) 000427[0] 1789367120.547110 trunc from 0x38fbe0 to 0x60a44e (0x27a86f bytes) 000431[0] 1789367120.603976 write 0x4f175e thru 0x4f1962 (0x205 bytes) 000432[1] 1789367120.639971 mapwrite 0x934765 thru 0x93607e (0x191a bytes) 000433[0] 1789367120.753440 write 0x2f4d90 thru 0x2f6768 (0x19d9 bytes) 000434[1] 1789367120.807969 mapwrite 0x3f8af0 thru 0x3f97ee (0xcff bytes) 000435[1] 1789367120.897356 mapread 0x787f55 thru 0x78848e (0x53a bytes) 000436[1] 1789367120.921927 trunc from 0x93607f to 0x2ad339 (0x688d45 bytes) 000438[1] 1789367121.036344 mapwrite 0x5223b2 thru 0x523f59 (0x1ba8 bytes) 000439[1] 1789367121.140655 write 0x2b6e95 thru 0x2b8a04 (0x1b70 bytes) 000442[1] 1789367121.202659 mapwrite 0x094f5b thru 0x0951b3 (0x259 bytes) 000443[0] 1789367121.275053 trunc from 0x523f5a to 0x43b5bc (0xe899d bytes) 000444[1] 1789367121.349736 read 0x156f43 thru 0x15846f (0x152d bytes) 000445[0] 1789367121.371607 read 0x32b148 thru 0x32c383 (0x123c bytes) 000446[0] 1789367121.387211 write 0x431c65 thru 0x43356b (0x1907 bytes) 000447[1] 1789367121.396697 punch from 0x25397d to 0x255623 (0x1ca7 bytes) 000448[0] 1789367121.438215 read 0x3f7aa4 thru 0x3f8e70 (0x13cd bytes) 000450[0] 1789367121.464284 read 0x335ecf thru 0x3360ae (0x1e0 bytes) 000452[0] 1789367121.490669 write 0x030376 thru 0x03111d (0xda8 bytes) 000454[0] 1789367121.533528 trunc from 0x43b5bd to 0x0f75e9 (0x343fd3 bytes) 000456[1] 1789367121.594599 read 0x071684 thru 0x07225c (0xbd9 bytes) 000457[0] 1789367121.647466 punch from 0x0574a0 to 0x0579d6 (0x537 bytes) 000459[1] 1789367121.692383 mapread 0x06b16b thru 0x06bb28 (0x9be bytes) 000460[0] 1789367121.734640 punch from 0x08351a to 0x084e38 (0x191f bytes) 000464[1] 1789367121.810355 mapread 0x04fd66 thru 0x051be1 (0x1e7c bytes) 000465[0] 1789367121.869719 write 0x433c12 thru 0x4355b4 (0x19a3 bytes) 000466[1] 1789367121.908395 punch from 0x3141ca to 0x3160ec (0x1f23 bytes) 000468[0] 1789367121.966218 punch from 0x2a30a7 to 0x2a4911 (0x186b bytes) 000471[1] 1789367121.998101 read 0x3a2c5a thru 0x3a3553 (0x8fa bytes) 000474[0] 1789367122.030329 punch from 0x0ea472 to 0x0ebfbe (0x1b4d bytes) 000475[0] 1789367122.049334 mapwrite 0x32a26c thru 0x32c1b5 (0x1f4a bytes) 000476[1] 1789367122.124331 trunc from 0x4355b5 to 0x40b05f (0x2a555 bytes) 000478[1] 1789367122.204316 write 0x9f4024 thru 0x9f5c1e (0x1bfb bytes) 000481[1] 1789367122.235349 read 0x2d6ee6 thru 0x2d729a (0x3b5 bytes) 000482[1] 1789367122.264357 read 0x50c517 thru 0x50d45e (0xf48 bytes) 000483[1] 1789367122.273188 read 0x8bb801 thru 0x8bc29d (0xa9d bytes) 000484[0] 1789367122.278155 punch from 0x9d50de to 0x9d6208 (0x112b bytes) 000485[0] 1789367122.316034 read 0x2a7c4e thru 0x2a8413 (0x7c6 bytes) 000486[0] 1789367122.334268 mapread 0x2601e0 thru 0x260514 (0x335 bytes) 000489[1] 1789367122.369188 write 0x50cc3e thru 0x50e4be (0x1881 bytes) 000491[0] 1789367122.404696 punch from 0x68ce7b to 0x68d168 (0x2ee bytes) 000492[1] 1789367122.453135 write 0x7ff6a4 thru 0x8012ed (0x1c4a bytes) 000494[1] 1789367122.538636 mapread 0x00d4ff thru 0x00eb2d (0x162f bytes) 000495[0] 1789367122.576439 read 0x97518e thru 0x97573c (0x5af bytes) 000497[1] 1789367122.655846 read 0x59b65f thru 0x59c401 (0xda3 bytes) 000499[0] 1789367122.747872 punch from 0x7ee0a2 to 0x7efb8c (0x1aeb bytes) 000500[0] 1789367122.797449 mapwrite 0x6bc359 thru 0x6bccd7 (0x97f bytes) All operations completed A-OK! PASS 16b (27s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:25:27 (1789367127) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:25:31 (1789367131) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.353767 s, 23.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.329814 s, 25.4 MB/s ldlm.namespaces.MGC192.168.201.110@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d628319d000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8d62a0d23000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d628319d000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d628319d000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63936 s, 10.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.89552 s, 36.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.18509 s, 48.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.73637 s, 22.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.5628 s, 16.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.27899 s, 32.0 MB/s PASS 16d (41s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:26:12 (1789367172) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286893 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998713 s, 10.5 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:26:19 (1789367179) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:26:45 (1789367205) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 02:27:12 (1789367232) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 02:27:18 (1789367238) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 02:27:25 (1789367245) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.26133 s, 20.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.98741 s, 26.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.8885 s, 13.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.10658 s, 23.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50524 s, 17.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.60367 s, 16.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98799 s, 13.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.88079 s, 13.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0029 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07932 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.22929 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.8914 s, 13.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98443 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43551 s, 10.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.48092 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.98856 s, 13.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.17117 s, 12.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43795 s, 10.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.71641 s, 9.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (30s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:27:55 (1789367275) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Sep 14 02:28:31 EDT 2026 + fsx 2 done: rc=0 Mon Sep 14 02:28:31 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 3 done: rc=0 Mon Sep 14 02:28:32 EDT 2026 + fsx 4 done: rc=0 Mon Sep 14 02:28:32 EDT 2026 + fsx 5 done: rc=0 Mon Sep 14 02:28:32 EDT 2026 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:28:38 (1789367318) fail_loc=0x8000030a PASS 17 (8s) == sanityn test 18: mmap sanity check =========================================================================================== 02:28:46 (1789367326) mmap test1: basic mmap operation (PASS, 0.10251s) mmap test2: MAP_PRIVATE not write back (PASS, 0.095062s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.741s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4289s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.14572s) mmap test6: check mmap write/read content on two nodes (PASS, 0.16641s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.83831s) mmap test11: multi-thread shared mmap (PASS, 1.0068s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1006s) PASS 18 (25s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:29:11 (1789367351) oleg110-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:29:15 (1789367355) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:29:21 (1789367361) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 02:29:26 (1789367366) now is 1789367368 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 new atime is 1789367429 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:30:33 (1789367433) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:30:39 (1789367439) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:30:45 (1789367445) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:30:50 (1789367450) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 02:30:53 (1789367453) utime: good mknod times 1789367453 <= 1789367454 <= 1789367454 for /mnt/lustre/f26a utime: good mknod times 1789367453 <= 1789367454 <= 1789367454 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 02:30:59 (1789367459) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:31:07 (1789367467) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-14 02:31:09.000000000 -0400 Birth: 2026-09-14 02:31:09.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-14 02:31:09.000000000 -0400 Birth: 2026-09-14 02:31:09.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:31:15 (1789367475) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:31:23 (1789367483) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:31:30 (1789367490) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.110@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (19s) == sanityn test 31r: open-rename(replace) race =========== 02:31:49 (1789367509) -rw-r--r-- 1 root root 0 Sep 14 02:31 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:31:58 (1789367518) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43148 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:32:04 (1789367524) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 02:32:11 (1789367531) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 02:32:14 (1789367534) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.24933 s, 42.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.918154 s, 57.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51949 s, 34.5 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.68583 s, 31.1 MB/s contention_seconds 4, contention_events 21 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48038 s, 35.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63798 s, 32.0 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.00117 s, 52.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.982827 s, 53.3 MB/s PASS 32c (24s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:32:40 (1789367560) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:32:43 (1789367563) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:32:46 (1789367566) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:32:49 (1789367569) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 02:32:52 (1789367572) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid in IDLE state after 0 sec oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid in FULL state after 0 sec oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid in IDLE state after 0 sec oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid in FULL state after 0 sec oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d628319d000.ost_server_uuid in IDLE state after 0 sec oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d628319d000.ost_server_uuid in FULL state after 0 sec PASS 34 (118s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 02:34:50 (1789367690) fail_loc=0x80000317 Race attempt 0 Wait for 50450 50523 for 60 sec... - open/close 908 (time 1789367705.90 total 10.00 last 90.76) - open/close 1860 (time 1789367715.90 total 20.01 last 95.18) - open/close 2844 (time 1789367725.91 total 30.01 last 98.35) - open/close 3728 (time 1789367735.91 total 40.02 last 88.33) total: 4000 open/close in 43.34 seconds: 92.29 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.110@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.201.110@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 02:36:01 (1789367761) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg110-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08258 s, 48.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.964864 s, 54.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.936445 s, 56.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.793105 s, 66.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.825226 s, 63.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.793143 s, 66.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.611774 s, 85.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.619966 s, 84.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.655095 s, 80.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.745745 s, 70.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.684568 s, 76.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7493632) after_dd(7442432) after(7493632) PASS 36 (286s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 02:40:47 (1789368047) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 - create 2147 (time 1789368060.19 total 10.01 last 214.51) - create 4104 (time 1789368070.19 total 20.01 last 195.69) - create 6031 (time 1789368080.19 total 30.02 last 192.58) - create 7854 (time 1789368090.19 total 40.02 last 182.29) - create 9755 (time 1789368100.20 total 50.02 last 189.98) total: 10000 create in 51.24 seconds: 195.16 ops/second PASS 37 (59s) == sanityn test 39a: file mtime does not change after rename ========================================================== 02:41:46 (1789368106) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 02:41:51 (1789368111) repeat after cancel_lru_locks PASS 39b (6s) == sanityn test 39c: check truncate mtime update ================================================================================ 02:41:57 (1789368117) repeat after cancel_lru_locks PASS 39c (5s) == sanityn test 39d: sync write should update mtime ====== 02:42:02 (1789368122) fail_loc=0x411 fail_loc=0 PASS 39d (5s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 02:42:08 (1789368128) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (12s) == sanityn test 40c: pdirops: link and others ======================================================================== 02:42:20 (1789368140) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (12s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 02:42:32 (1789368152) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (11s) == sanityn test 40e: pdirops: rename and others ======================================================================== 02:42:43 (1789368163) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (11s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 02:42:55 (1789368175) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (8s) == sanityn test 41b: pdirops: create vs create ======================================================================== 02:43:03 (1789368183) open: File exists Conflict PASS 41b (8s) == sanityn test 41c: pdirops: create vs link ======================================================================== 02:43:11 (1789368191) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (9s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 02:43:20 (1789368200) Conflict PASS 41d (8s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 02:43:28 (1789368208) Conflict PASS 41e (8s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 02:43:36 (1789368216) Conflict PASS 41f (9s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 02:43:45 (1789368225) Conflict PASS 41g (8s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 02:43:53 (1789368233) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (8s) == sanityn test 41i: reint_open: create vs create ======== 02:44:01 (1789368241) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (818s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:57:39 (1789369059) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (7s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:57:46 (1789369066) open: File exists Conflict PASS 42b (6s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:57:52 (1789369072) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (7s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:57:59 (1789369079) Conflict PASS 42d (8s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:58:07 (1789369087) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (7s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:58:14 (1789369094) Conflict PASS 42f (7s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:58:21 (1789369101) Conflict PASS 42g (7s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:58:28 (1789369108) Conflict PASS 42h (7s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:58:35 (1789369115) PASS 43a (36s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:59:11 (1789369151) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:59:17 (1789369157) Conflict PASS 43c (7s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:59:24 (1789369164) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (7s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:59:31 (1789369171) Conflict PASS 43e (7s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:59:38 (1789369178) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (8s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:59:46 (1789369186) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (7s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:59:53 (1789369193) Conflict PASS 43h (8s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:00:01 (1789369201) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:00:03 (1789369203) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (62s) == sanityn test 43k: unlink vs create ==================== 03:01:04 (1789369264) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (474s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 03:08:58 (1789369738) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Sep 14 03:09:02 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 03:09:04 (1789369744) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 03:09:09 (1789369749) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 03:09:14 (1789369754) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 03:09:19 (1789369759) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 03:09:25 (1789369765) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 03:09:30 (1789369770) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 03:09:36 (1789369776) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 03:09:42 (1789369782) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 03:09:43 (1789369783) PASS 45a (37s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 03:10:20 (1789369820) Conflict PASS 45b (6s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 03:10:26 (1789369826) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 03:10:32 (1789369832) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 03:10:37 (1789369837) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 03:10:43 (1789369843) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 03:10:48 (1789369848) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 03:10:53 (1789369853) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 03:10:57 (1789369857) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 03:10:58 (1789369858) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (429s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:18:07 (1789370287) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:18:12 (1789370292) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:18:17 (1789370297) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:18:22 (1789370302) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:18:28 (1789370308) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:18:33 (1789370313) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:18:38 (1789370318) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:18:43 (1789370323) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:18:48 (1789370328) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:18:49 (1789370329) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:18:50 (1789370330) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:18:51 (1789370331) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:18:52 (1789370332) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:18:53 (1789370333) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:18:54 (1789370334) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:18:55 (1789370335) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:18:56 (1789370336) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00581109 s, 1.8 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:19:03 (1789370343) 0+1 records in 0+1 records out 158 bytes copied, 0.00351796 s, 44.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (219017) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:19:07 (1789370347) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00522239 s, 196 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160745 s, 65.2 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:19:22 (1789370362) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00385309 s, 266 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:19:28 (1789370368) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167302 s, 62.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:19:31 (1789370371) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 03:19:35 (1789370375) llite.lustre-ffff8d628319d000.xattr_cache=1 llite.lustre-ffff8d62a0d23000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:36.000000000 -0400 Modify: 2026-09-14 03:19:36.000000000 -0400 Change: 2026-09-14 03:19:36.000000000 -0400 Birth: 2026-09-14 03:19:36.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:41.000000000 -0400 Modify: 2026-09-14 03:19:42.000000000 -0400 Change: 2026-09-14 03:19:42.000000000 -0400 Birth: 2026-09-14 03:19:41.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:47.000000000 -0400 Modify: 2026-09-14 03:19:47.000000000 -0400 Change: 2026-09-14 03:19:47.000000000 -0400 Birth: 2026-09-14 03:19:47.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:47.000000000 -0400 Modify: 2026-09-14 03:19:47.000000000 -0400 Change: 2026-09-14 03:19:47.000000000 -0400 Birth: 2026-09-14 03:19:47.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:52.000000000 -0400 Modify: 2026-09-14 03:19:52.000000000 -0400 Change: 2026-09-14 03:19:52.000000000 -0400 Birth: 2026-09-14 03:19:52.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 03:19:52.000000000 -0400 Modify: 2026-09-14 03:19:52.000000000 -0400 Change: 2026-09-14 03:19:52.000000000 -0400 Birth: 2026-09-14 03:19:52.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 03:19:59 (1789370399) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 03:20:07 (1789370407) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 03:20:15 (1789370415) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 03:20:27 (1789370427) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:20:37 (1789370437) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:20:38 (1789370438) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 03:20:51 (1789370451) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:21:05 (1789370465) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789370466 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.65387 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (26s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:21:31 (1789370491) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789370493 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.59042 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (39s) == sanityn test 60: Verify data_version behaviour ======== 03:22:10 (1789370530) PASS 60 (3s) == sanityn test 70a: cd directory && rm directory ======== 03:22:13 (1789370533) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:22:16 (1789370536) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:22:18 (1789370538) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0027884 s, 23.5 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:22:19 (1789370539) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00347657 s, 18.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:22:21 (1789370541) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:22:22 (1789370542) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00522791 s, 12.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:22:23 (1789370543) llite.lustre-ffff8d628319d000.xattr_cache=1 llite.lustre-ffff8d62a0d23000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:22:25 (1789370545) llite.lustre-ffff8d628319d000.xattr_cache=1 llite.lustre-ffff8d62a0d23000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:22:27 (1789370547) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00299362 s, 342 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00328705 s, 312 kB/s 231784: get lock2 231785: get lock1 231784: done 231784: get lock1 231785: done 231785: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231784: cannot get lock1: Resource deadlock avoided 231785: done 231785: exit rc=0 231784: done 231784: exit rc=0 PASS 74 (7s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:22:34 (1789370554) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0258163 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111907 s, 93.7 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232366 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 03:22:41 (1789370561) opening files: ........ closing files: ........ PASS 76 (21s) == sanityn test 77a: check FIFO NRS policy =============== 03:23:02 (1789370582) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.130987 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295465 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520487 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664126 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512417 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105641 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137386 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129945 s, 8.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129865 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138076 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158297 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155467 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138521 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160698 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151232 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139516 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178114 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129061 s, 81.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990279 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248756 s, 42.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214119 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272569 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257303 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00700968 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282492 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345589 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798372 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407777 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283053 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00649573 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00401466 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412484 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00483149 s, 217 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (3s) == sanityn test 77b: check CRR-N NRS policy ============== 03:23:05 (1789370585) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134347 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054271 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521775 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916124 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655939 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973476 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1055 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13065 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112424 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123309 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150226 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121843 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153357 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110693 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115409 s, 9.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132593 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213452 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00713783 s, 147 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00697819 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259992 s, 40.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180584 s, 58.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330976 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309648 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204538 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00754719 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343975 s, 30.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314785 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014899 s, 70.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410705 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300059 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259285 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204813 s, 51.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.142821 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330775 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387776 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651837 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767275 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889177 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740046 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865947 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105874 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104155 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895009 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1168 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132907 s, 7.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12086 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122723 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128203 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122899 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148404 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174268 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159382 s, 65.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151545 s, 69.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017803 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368608 s, 284 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023878 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189804 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217285 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153483 s, 68.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00575413 s, 182 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940072 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125766 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104525 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422632 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310942 s, 33.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (5s) == sanityn test 77c: check ORR NRS policy ================ 03:23:10 (1789370590) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124686 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341847 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066957 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717092 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704442 s, 14.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105847 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148571 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166411 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01639 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303849 s, 34.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243261 s, 43.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311135 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477276 s, 22.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19478 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337877 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184044 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306473 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021263 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321659 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154865 s, 67.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170974 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19304 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021416 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206554 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198134 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222218 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206743 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474272 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129578 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209078 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556857 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679661 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244388 s, 4.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138125 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504741 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747609 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545786 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837397 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807903 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843826 s, 12.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966749 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852435 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900186 s, 11.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965922 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159827 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152905 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154783 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151191 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168642 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182465 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299689 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194226 s, 54.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199627 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026321 s, 39.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032268 s, 32.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225064 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01779 s, 58.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953063 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162604 s, 64.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405297 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460008 s, 22.8 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112727 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157099 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00912703 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204187 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386941 s, 27.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.132424 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286746 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737833 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852855 s, 12.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724017 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108799 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101439 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986637 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114077 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989132 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117727 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903568 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122924 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11103 s, 9.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103252 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123624 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128347 s, 8.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00940283 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103295 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228742 s, 45.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165516 s, 63.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416694 s, 252 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101735 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266402 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163078 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251025 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219214 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291802 s, 35.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278106 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254221 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220264 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011122 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157292 s, 66.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 03:23:15 (1789370595) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143955 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03745 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983708 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115746 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119551 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118347 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140489 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135047 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119655 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110346 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13191 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127892 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17644 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141104 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15246 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148524 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14289 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173985 s, 60.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290861 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300044 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190392 s, 55.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284883 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168033 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215194 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182788 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013748 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141339 s, 74.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196347 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135941 s, 77.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105739 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676555 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377481 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654473 s, 160 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145965 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246084 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441446 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385277 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619985 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779615 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557486 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105269 s, 10.0 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101471 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864595 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117912 s, 8.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963514 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101827 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128868 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136294 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187211 s, 56.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212602 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263399 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294375 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137231 s, 76.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347679 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144594 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122414 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151325 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406854 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679816 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962045 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043571 s, 241 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117633 s, 89.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00887824 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262122 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00682748 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0052955 s, 198 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155136 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424446 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582763 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823317 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797934 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855218 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950012 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790236 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885478 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973351 s, 10.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121565 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158559 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156327 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150919 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156732 s, 6.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142946 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160792 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322646 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028687 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337804 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300296 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253712 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327559 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239944 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022164 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179983 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209423 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211713 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00528087 s, 199 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404826 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039492 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00676461 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725405 s, 145 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 03:23:20 (1789370600) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.201.10@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127213 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292338 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449327 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825879 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100042 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798332 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103102 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773592 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963287 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872457 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10455 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903702 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107462 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11048 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100032 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101578 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099874 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118043 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00716508 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198789 s, 52.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198813 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205347 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241242 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199905 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218668 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242585 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207812 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020218 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232683 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430553 s, 244 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00495026 s, 212 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402393 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00405222 s, 259 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.167127 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343493 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418121 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284851 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483244 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277859 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323087 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930747 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605867 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509757 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928794 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100388 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119968 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118232 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153857 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132578 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155511 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1296 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124532 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127305 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127752 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122929 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123018 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131688 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12719 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131353 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12583 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125009 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133209 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132233 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142795 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128353 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140186 s, 7.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.177292 s, 94.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474544 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529165 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709353 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743092 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755197 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808988 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067417 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880967 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927047 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104367 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101143 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112313 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110436 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122753 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12585 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138643 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174617 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132732 s, 79.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163445 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151914 s, 69.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196435 s, 53.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113809 s, 92.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168557 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137231 s, 76.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00875856 s, 120 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0090932 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230516 s, 45.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192503 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138932 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125695 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112632 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782955 s, 134 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.172706 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449558 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322501 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656625 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120881 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121354 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138823 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12768 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114885 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127906 s, 8.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145587 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129224 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138231 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139845 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121501 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12586 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128781 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157758 s, 66.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172636 s, 60.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188291 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210993 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018585 s, 56.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106492 s, 98.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177486 s, 59.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00817869 s, 128 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151526 s, 69.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295632 s, 35.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142695 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012684 s, 82.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243691 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00550945 s, 190 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00506305 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157307 s, 66.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (9s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:23:29 (1789370609) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.162927 s, 103 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401264 s, 26.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673241 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765517 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107379 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112555 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108431 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995373 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100259 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965462 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901548 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094133 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112247 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917239 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12017 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103157 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120495 s, 8.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153783 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221924 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211414 s, 49.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173366 s, 60.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197099 s, 53.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00424257 s, 247 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189568 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293347 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293436 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016026 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127317 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115719 s, 90.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267457 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00881105 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679781 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134294 s, 78.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.140495 s, 119 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293765 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541879 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341592 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605652 s, 17.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747165 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484985 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504024 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582349 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598359 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10618 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107339 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124434 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105254 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134868 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144652 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138675 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133533 s, 78.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172579 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019355 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199581 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138544 s, 75.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209004 s, 50.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242954 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227931 s, 46.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181944 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138302 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276438 s, 37.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113842 s, 92.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024569 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205458 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946709 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112594 s, 93.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135591 s, 124 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294595 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303258 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529868 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111687 s, 9.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132002 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155619 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141305 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1739 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195522 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180913 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175401 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18682 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212232 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190176 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214198 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204154 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215982 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027645 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038267 s, 27.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333778 s, 31.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371538 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269473 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503248 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034202 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360096 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430174 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425499 s, 24.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385153 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505448 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552093 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409693 s, 25.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056006 s, 18.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155818 s, 108 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315231 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023468 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654757 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552315 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772688 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629039 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735804 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718528 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528308 s, 19.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740162 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11383 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127918 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915605 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119854 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13326 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128722 s, 8.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163116 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135785 s, 77.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277893 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219013 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113671 s, 92.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292134 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259115 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302351 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415234 s, 253 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260765 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275295 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163337 s, 64.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651633 s, 161 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00773587 s, 136 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112807 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015462 s, 67.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (8s) == sanityn test 77g: Change TBF type directly ============ 03:23:37 (1789370617) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:23:40 (1789370620) oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 03:23:44 (1789370624) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:23:52 (1789370632) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14745 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230722 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041761 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907814 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950964 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968655 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899728 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104319 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815101 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920285 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871233 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102551 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823346 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996127 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912594 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111412 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918761 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0043517 s, 241 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124922 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188679 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168586 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176036 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159982 s, 65.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270686 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571919 s, 183 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313994 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627155 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335528 s, 31.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970553 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320898 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420869 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00262849 s, 399 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00283629 s, 370 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93112 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4579 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (55s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:24:48 (1789370688) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26229 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069959 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072693 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595265 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119264 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138416 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136313 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124584 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123104 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144623 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982572 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712555 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957601 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140578 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927769 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178673 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204131 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162152 s, 64.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290788 s, 36.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297493 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301906 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173311 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204011 s, 51.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242792 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394255 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405394 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321181 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325759 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333384 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341794 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341318 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391501 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427514 s, 24.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7376 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9358 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.23466 s, 13.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623851 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557353 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744559 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585132 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588092 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077869 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738596 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076176 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155181 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155146 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17625 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164515 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169771 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138793 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175842 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158058 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305709 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284011 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230866 s, 45.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323254 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024317 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356108 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379179 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400502 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324244 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025619 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384914 s, 27.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318481 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299582 s, 35.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342058 s, 30.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315964 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326277 s, 32.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.839 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.8182 s, 5.0 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (148s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:27:16 (1789370836) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3194 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816467 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129091 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214108 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22134 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265766 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247342 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231156 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181417 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180214 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274545 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239025 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335924 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299674 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321169 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259008 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.534462 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117113 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900563 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125891 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123402 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147085 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114968 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150804 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146869 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141061 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196738 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164901 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1688 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13074 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163665 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162124 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207002 s, 5.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8886 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9287 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2819 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838721 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945473 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972109 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119557 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137509 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964112 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116486 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15667 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941878 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141688 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23663 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225126 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233173 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250381 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242332 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248073 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706956 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749711 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689525 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654735 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746463 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886428 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712446 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101139 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784418 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747521 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754569 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928699 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104317 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951296 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887991 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10958 s, 9.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8969 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9313 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (150s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:29:46 (1789370986) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.417766 s, 40.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08715 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903544 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106041 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225437 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161298 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237887 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198811 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247129 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220587 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25062 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303345 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269192 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255667 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320799 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269136 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315332 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396279 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348606 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309943 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328816 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359905 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505128 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345901 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396996 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263003 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726361 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464398 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492236 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474554 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216341 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494955 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567325 s, 18.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.03277 s, 17.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89336 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.10@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.10@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.358225 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481415 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586247 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119934 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143828 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130054 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152164 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165691 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190496 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200794 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152715 s, 6.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151619 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222894 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203044 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188437 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202494 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214053 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00999353 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02643 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142407 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571881 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621034 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255327 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562644 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577966 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628509 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287763 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061689 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325033 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347536 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387376 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394172 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403337 s, 26.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.98266 s, 17.5 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85938 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.10@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.410377 s, 40.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370322 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389916 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112907 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066521 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110262 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968391 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722963 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444592 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994071 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13957 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195204 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192789 s, 5.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148141 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19611 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213515 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183284 s, 5.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039249 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683238 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042217 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557294 s, 188 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00597569 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048768 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062914 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707855 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842969 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427062 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0096899 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029455 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616308 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612143 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225026 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102833 s, 102 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34036 s, 19.6 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97887 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.294894 s, 56.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755076 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311065 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419696 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109277 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130213 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123015 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734852 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153076 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162624 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160742 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178713 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165054 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187492 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158971 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180687 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177892 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00880236 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262095 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451998 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017398 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176201 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274638 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517107 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226779 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415166 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332892 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021015 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229589 s, 45.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402203 s, 26.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421529 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639981 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106625 s, 98.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96513 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85608 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.65956 s, 18.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81629 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.309728 s, 54.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418178 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467399 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680634 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140193 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132139 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15133 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150529 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154448 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172918 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149608 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174657 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168404 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191199 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161303 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194873 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23414 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758199 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734664 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632447 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576324 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536502 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831882 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725551 s, 14.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729868 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069109 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677791 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837901 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893937 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801509 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796874 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839661 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091949 s, 11.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.704 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5254 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.356254 s, 47.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474245 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02554 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353529 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629552 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495951 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694439 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10943 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13297 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147115 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174651 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192779 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19451 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188378 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197381 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199904 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208104 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124968 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308281 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153225 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025554 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594281 s, 176 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173751 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297631 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308597 s, 34.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249463 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271172 s, 38.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236746 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243564 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278328 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287938 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313333 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352625 s, 29.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.34735 s, 19.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81032 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.478167 s, 35.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516671 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409396 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519035 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868606 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082428 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1085 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999629 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108052 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132093 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131314 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127509 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142991 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145546 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140325 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207504 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216487 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232839 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267422 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160535 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144788 s, 72.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146617 s, 71.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180337 s, 58.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169714 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188707 s, 55.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326113 s, 32.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218377 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220143 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201734 s, 52.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261445 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224437 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245325 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284143 s, 36.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.85405 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78442 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8169 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7789 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (396s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:36:22 (1789371382) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279715 s, 60.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271625 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299768 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530208 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104488 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847105 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116946 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107455 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119243 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115571 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110017 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135656 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126718 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114862 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133989 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128922 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137189 s, 7.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174792 s, 60.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218333 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255317 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282489 s, 37.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227118 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245592 s, 42.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284806 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031203 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252787 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322189 s, 32.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272537 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290968 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286217 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286365 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303351 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307319 s, 34.1 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95997 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.75545 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (39s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:37:01 (1789371421) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.201.10@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.201.10@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.450096 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586935 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607775 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117513 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245452 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315046 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343396 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352238 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296645 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296729 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334772 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331395 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334602 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346249 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330167 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320973 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384069 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216718 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173893 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222469 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175919 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183288 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157161 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169914 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18092 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157332 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193017 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164095 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206423 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188972 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19636 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192235 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235937 s, 4.4 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5831 s, 8.3 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.172 s, 8.6 MB/s Read runtime is 14 s, speed is 7.142857 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (73s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:38:14 (1789371494) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.201.10@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27225 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112846 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144625 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240036 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27358 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184956 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387172 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349437 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447265 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472537 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432334 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487844 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508134 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479931 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.53123 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475712 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.648141 s, 1.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692401 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107672 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100024 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930518 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880897 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572175 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065158 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11697 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511858 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792761 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906453 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669343 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991576 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718125 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741282 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106045 s, 9.9 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 15.4327 s, 6.8 MB/s Write runtime is 17 s, speed is 5.882352 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.46978 s, 19.2 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (60s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:39:14 (1789371554) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.561284 s, 29.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587976 s, 17.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737342 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077585 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806931 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126246 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125872 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104768 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113265 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118319 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186699 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828555 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160849 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152434 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13276 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31969 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378526 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148072 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560335 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364145 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214519 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397091 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397018 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656089 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477298 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427805 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540025 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541144 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049007 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566429 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421926 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421777 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569951 s, 18.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7205 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91668 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.58415 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0109 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (97s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:40:51 (1789371651) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03118 s, 16.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826116 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465759 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666659 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103939 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189406 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207118 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238291 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264184 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2517 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275133 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280503 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31308 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315832 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346279 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369078 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384208 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036759 s, 28.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864674 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270607 s, 38.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851411 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074804 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835346 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890099 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050144 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828948 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667398 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100927 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802143 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768648 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784564 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895885 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883383 s, 11.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7571 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0318 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (81s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:42:13 (1789371733) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.693741 s, 24.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512951 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601441 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522545 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782086 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1478 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187802 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173994 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247772 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26974 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286679 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282896 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277395 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280664 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287493 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271588 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8d62a0d23000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358808 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151582 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100368 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16499 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121348 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126203 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127197 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122464 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137824 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139968 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139459 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146137 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200782 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158792 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141858 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15262 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227345 s, 4.6 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.36189 s, 12.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92715 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg110-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.99464 s, 16.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943078 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072621 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140845 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129179 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183217 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174733 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191351 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243321 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374536 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360293 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454523 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480607 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518215 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.539555 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41192 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.574075 s, 1.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62a0d23000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875995 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394164 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491204 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509229 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127722 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592895 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696724 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076937 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114526 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195321 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175576 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136047 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00783515 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551818 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195426 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260966 s, 40.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8717 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89944 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.8944 s, 8.1 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9164 s, 10.6 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (148s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:44:41 (1789371881) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:) Stopping client oleg110-client.virtnet /mnt/lustre opts: Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg110-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg110-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg110-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.10,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg110-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.10,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg110-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.10,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70208/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.201.10,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg110-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Starting client oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Started clients oleg110-client.virtnet: 192.168.201.110@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Started clients oleg110-client.virtnet: 192.168.201.110@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d62837a1000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d6298c0e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d62837a1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d6298c0e000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618488 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886042 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911054 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173331 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206142 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207135 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138593 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221626 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211806 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151126 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248885 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141226 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363068 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268147 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280838 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386574 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418667 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352419 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331072 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363936 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367538 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359155 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362987 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374351 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386866 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431268 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403852 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424225 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.430081 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453996 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507363 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432848 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476493 s, 2.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.09907 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.44188 s, 19.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.681216 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124568 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140103 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773597 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850545 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958799 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124666 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149801 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747862 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141608 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193598 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170886 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266679 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227156 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307737 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278138 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340308 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107971 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710351 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758882 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981517 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782139 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736219 s, 14.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791031 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848425 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119009 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114505 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107017 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156707 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100109 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166555 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100017 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121687 s, 8.6 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.13106 s, 12.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0136 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (235s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 03:48:37 (1789372117) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (20s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 03:48:56 (1789372136) On MGS 192.168.201.110, active = nodemap.active=1 On MGS 192.168.201.110, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598006 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565173 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514431 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584427 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692794 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101174 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505743 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106032 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163798 s, 6.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820883 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142202 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137579 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1344 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210551 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176211 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153554 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188753 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610478 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431213 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775786 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336006 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463312 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542681 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141059 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570225 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892032 s, 11.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712666 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902252 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736015 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109413 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262333 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100171 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115322 s, 9.1 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.14694 s, 14.7 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.49685 s, 19.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.201.110, TBF.id = On MGS 192.168.201.110, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.795203 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18396 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262821 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257867 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307298 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461218 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408308 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46394 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.576355 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452407 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.592768 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447552 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.686749 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.550153 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477062 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.684547 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.594016 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138821 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923365 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946668 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904179 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664656 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098207 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836965 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192593 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146208 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131933 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188087 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162175 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14961 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155921 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140789 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244119 s, 4.3 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.85118 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.96376 s, 26.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg110-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 On MGS 192.168.201.110, active = nodemap.active=1 On MGS 192.168.201.110, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632678 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129115 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897925 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629152 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115103 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184696 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219062 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320754 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319498 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222418 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282395 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273279 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414823 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227944 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421877 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399286 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380477 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140605 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14206 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152362 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171099 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159279 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173993 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197957 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235152 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167131 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198127 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222512 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205008 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215853 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223336 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241195 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239735 s, 4.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4235 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0634 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.201.110, TBF.id = On MGS 192.168.201.110, active = nodemap.active=0 PASS 77kj (204s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:52:20 (1789372340) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (13s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:52:33 (1789372353) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg110-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.47692 s, 302 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg110-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1167 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (60s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:53:33 (1789372413) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.462649 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488057 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046104 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910199 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110743 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243339 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177255 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196275 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161735 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2887 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196531 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223865 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278363 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196263 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324094 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321251 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327846 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734773 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134656 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780269 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787363 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027403 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104454 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879024 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892724 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909364 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105178 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054665 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943406 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127471 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117394 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067344 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132124 s, 7.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg110-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg110-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.20096 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.42089 s, 19.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.531977 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420633 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10219 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105366 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22705 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242504 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283935 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315487 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340701 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296379 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340284 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377283 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356348 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351225 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391897 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378062 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356769 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8d62837a1000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918167 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708859 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469151 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107807 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345588 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065255 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581926 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00704767 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685747 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107923 s, 9.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8d62837a1000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555703 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06993 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00287291 s, 365 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111285 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814582 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185153 s, 56.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg110-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg110-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.95814 s, 15.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.39543 s, 19.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 03:55:04 (1789372504) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:55:17 (1789372517) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (133s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 03:57:30 (1789372650) mds.MDS.mdt.nrs_policies=tbf oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg110-client: oleg110-server: ssh exited with exit code 36 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg110-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg110-client: oleg110-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 03:58:08 (1789372688) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 274979 kill md_thread xargs PASS 77r (59s) == sanityn test 77s: Check TBF LRU shrinker ============== 03:59:07 (1789372747) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (72s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 04:00:19 (1789372819) oleg110-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg110-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg110-client: oleg110-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 04:00:30 (1789372830) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 04:00:32.000000000 -0400 Modify: 2026-09-14 04:00:32.000000000 -0400 Change: 2026-09-14 04:00:32.000000000 -0400 Birth: 2026-09-14 04:00:32.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 04:00:50 (1789372850) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 04:00:54 (1789372854) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 04:00:57 (1789372857) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 04:01:02 (1789372862) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:01:05 (1789372865) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:01:09 (1789372869) - open/close 826 (time 1789372885.98 total 10.03 last 82.39) - open/close 1666 (time 1789372895.98 total 20.03 last 83.98) total: 2500 open/close in 29.68 seconds: 84.22 ops/second snapshot_time 1789373002.251654299 secs.nsecs start_time 1789372872.845498235 secs.nsecs elapsed_time 129.406156064 secs.nsecs open 2505 samples [usecs] 135 17105 3397767 7818474635 close 2505 samples [usecs] 162 13647 873070 644329998 mknod 2501 samples [usecs] 557 15885 2982514 6156902388 unlink 2500 samples [usecs] 681 176896 7618010 164731368402 mkdir 2 samples [usecs] 1060 2711 3771 8473121 rmdir 2 samples [usecs] 4112 34570 38682 1211993444 rename 2500 samples [usecs] 447 53463 17377056 253671223566 getattr 3761 samples [usecs] 7 13388 253783 697726261 parallel_rename_file 2500 samples [usecs] 447 53463 17377056 253671223566 crossdir_rename 2500 samples [usecs] 447 53463 17377056 253671223566 readpage 33 samples [usecs] 6078 94805 1225972 66728375780 PASS 81d (139s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:03:28 (1789373008) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7614 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:03:37 (1789373017) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:03:40 (1789373020) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 04:03:52 (1789373032) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789373034 Finishing test test1 at 1789373037 PASS 85 (9s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:04:01 (1789373041) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 04:04:03 (1789373043) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:04:07 (1789373047) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:04:10 (1789373050) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5802 0x16aa 0x240000400 1 5762 0x1682 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5803 0x16ab 0x240000400 1 5763 0x1683 0x280000400 rc1=0 and rc2=0 PASS 93 (16s) == sanityn test 94: signal vs CP callback race =========== 04:04:26 (1789373066) osc.lustre-OST0000-osc-ffff8d62837a1000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8d6298c0e000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d62837a1000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8d6298c0e000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.253854 s, 32.3 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.96899 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 291793 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0957841 s, 42.8 kB/s osc.lustre-OST0000-osc-ffff8d62837a1000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8d6298c0e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d62837a1000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d6298c0e000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:04:41 (1789373081) llite.lustre-ffff8d62837a1000.fast_read=0 llite.lustre-ffff8d6298c0e000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0688265 s, 30.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8d62837a1000.fast_read=1 llite.lustre-ffff8d6298c0e000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:05:01 (1789373101) llite.lustre-ffff8d62837a1000.fast_read=0 llite.lustre-ffff8d6298c0e000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0152184 s, 807 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8d62837a1000.fast_read=1 llite.lustre-ffff8d6298c0e000.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:05:14 (1789373114) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:05:18 (1789373118) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00803747 s, 510 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:05:25 (1789373125) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119705 s, 17.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:05:32 (1789373132) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.246865 s, 8.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 04:05:39 (1789373139) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0043837 s, 2.7 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 04:05:46 (1789373146) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109676 s, 373 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 04:05:55 (1789373155) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0385733 s, 106 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 04:06:02 (1789373162) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0194208 s, 211 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 04:06:10 (1789373170) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 04:06:18 (1789373178) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789373180 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789373184 Finishing test test23 at 1789373184 PASS 103 (12s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:06:30 (1789373190) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 04:06:33 (1789373193) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 04:06:56 (1789373216) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:07:00 (1789373220) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0155741 s, 65.8 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 04:07:02.000000000 -0400 Modify: 2026-09-14 04:07:02.000000000 -0400 Change: 2026-09-14 04:07:02.000000000 -0400 Birth: 2026-09-14 04:07:02.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1789373222 1789373222 1789373222 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 04:07:02.000000000 -0400 Modify: 2026-09-14 04:07:02.000000000 -0400 Change: 2026-09-14 04:07:02.000000000 -0400 Birth: 2026-09-14 04:07:02.000000000 -0400 1789373222 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 04:07:09 (1789373229) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 04:07:16 (1789373236) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.440162 s, 23.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5812 0x16b4 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:07:26 (1789373246) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.182968 s, 57.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5770 0x168a 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7614 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 04:07:40 (1789373260) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00610128 s, 1.6 MB/s llite.lustre-ffff8d62837a1000.hybrid_io=0 llite.lustre-ffff8d6298c0e000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.7526 s, 1.8 MB/s PASS 108a (14s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:07:54 (1789373274) 192.168.201.110@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.110@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Iteration 1 Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 192.168.201.110@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.201.110@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre2 PASS 109 (82s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:09:16 (1789373356) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-14 04:09:18.000000000 -0400 Modify: 2026-09-14 04:09:18.000000000 -0400 Change: 2026-09-14 04:09:18.000000000 -0400 Birth: 2026-09-14 04:09:18.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg110-client.virtnet oleg110-client.virtnet PASS 110 (74s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:10:30 (1789373430) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 04:10:34 (1789373434) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 04:10:38 (1789373438) lustre-OST0000-osc-ffff8d62862e0000 active. lustre-OST0001-osc-ffff8d62862e0000 active. lustre-OST0000-osc-ffff8d62862e0000 active. lustre-OST0001-osc-ffff8d62862e0000 active. MGC192.168.201.110@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 04:10:45 (1789373445) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:10:50 (1789373450) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:10:54 (1789373454) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 04:10:58 (1789373458) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0276148 s, 742 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.112194 s, 183 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (10s) == sanityn test 122: directory size is consistent across mounts ========================================================== 04:11:08 (1789373468) PASS 122 (10s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:11:18 (1789373478) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.162 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.292 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2396 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2706 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3136 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2449 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2875 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1126 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2988 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2948 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3286 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2764 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3704 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3613 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.27 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3943 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.417 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6503 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg110-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8d62862e0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d62862e0000.ost_server_uuid in FULL state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 6950 sec ============== 04:12:47 (1789373567) === sanityn: start cleanup 04:12:49 (1789373569) === Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg110-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:16:41 (1789373801) ===