-----============= acceptance-small: sanityn ============----- Wed Aug 26 04:43:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg304-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 04:43:39 (1787733819) === Starting client oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Mount client oleg304-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Started clients oleg304-client.virtnet: 192.168.203.104@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg304-client.virtnet: executing check_config_client /mnt/lustre oleg304-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg304-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff977c47b58800.idle_timeout=debug osc.lustre-OST0000-osc-ffff977c517f0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c47b58800.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c517f0000.idle_timeout=debug disable quota as required oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 04:44:14 (1787733854) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385276 s, 27.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7645] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 04:44:16 (1787733856) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 04:44:25 (1787733865) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 04:44:32 (1787733872) -rw-r--r-- 1 root root 0 Aug 26 04:44 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 04:44:38 (1787733878) -rw-r--r-- 1 root root 0 Aug 26 04:44 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 04:44:44 (1787733884) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 04:44:50 (1787733890) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 04:44:57 (1787733897) total 1 -rwxrwxrwx 1 root root 0 Aug 26 04:44 f2a -rwxrwxrwx 1 root root 0 Aug 26 04:44 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 26 04:44 f2a -rwxrwxrwx 1 root root 0 Aug 26 04:44 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0129273 s, 39.6 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 04:45:03 (1787733903) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 04:45:06 (1787733906) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.405551 s, 5.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 04:45:13 (1787733913) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 04:45:20 (1787733920) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 04:45:28 (1787733928) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 04:45:34 (1787733934) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 04:45:40 (1787733940) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 04:45:46 (1787733946) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 04:45:52 (1787733952) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 04:46:00 (1787733960) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792232 s, 13.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0552798 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 04:46:06 (1787733966) 1+0 records in 1+0 records out 1 byte copied, 0.00543986 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0107597 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00930432 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00818579 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0129466 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00934267 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00990416 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0126694 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010475 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110081 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0145097 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.008408 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 04:46:14 (1787733974) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0228789 s, 134 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0311612 s, 131 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00089322 s, 3.4 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 04:46:19 (1787733979) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 04:46:26 (1787733986) warning: '-runtime' deprecated, use '-t runtime' instead - link 418 (time 1787733998.05 total 10.00 last 41.80) - link 749 (time 1787734008.05 total 20.00 last 33.10) - link 1046 (time 1787734018.08 total 30.03 last 29.62) - link 1325 (time 1787734028.09 total 40.04 last 27.87) - link 1656 (time 1787734038.10 total 50.05 last 33.07) - link 1992 (time 1787734048.14 total 60.09 last 33.46) - link 2352 (time 1787734058.19 total 70.14 last 35.83) - link 2699 (time 1787734068.22 total 80.18 last 34.57) - link 3056 (time 1787734078.24 total 90.19 last 35.64) total: 3430 link in 100.03 seconds: 34.29 ops/second using seed 3387683663 running for 100 seconds - stat 6863 (time 1787734000 ; total 11 ; last 11) - stat 10000 (time 1787734006 ; total 17 ; last 6) - stat 14756 (time 1787734017 ; total 28 ; last 11) - stat 19562 (time 1787734028 ; total 39 ; last 11) - stat 20000 (time 1787734028 ; total 39 ; last 0) - stat 25780 (time 1787734039 ; total 50 ; last 11) - stat 28411 (time 1787734050 ; total 61 ; last 11) - stat 30000 (time 1787734056 ; total 67 ; last 6) - stat 32850 (time 1787734067 ; total 78 ; last 11) - stat 35928 (time 1787734078 ; total 89 ; last 11) - stat 38774 (time 1787734089 ; total 100 ; last 11) total: 38775 stats in 100 seconds: 387.750000 stats/second - unlinked 0 (time 1787734039 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3430) error: No such file or directory total: 3429 unlinks in 90 seconds: 38.099998 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21446) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21448) - No such process PASS 12 (149s) == sanityn test 13: test directory page revocation ======= 04:48:55 (1787734135) f13 f13 PASS 13 (7s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 04:49:02 (1787734142) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 22636 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (6s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 04:49:09 (1787734149) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 23215 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 04:49:15 (1787734155) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23793 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 04:49:21 (1787734161) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24377 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 04:49:27 (1787734167) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24959 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 04:49:35 (1787734175) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (49s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 04:50:24 (1787734224) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.498548 s, 16.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.545534 s, 15.4 MB/s Chance of close/open is 1 in 50 Seed set to 4228 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005[0] 1787734229.133181 write 0x5af1be thru 0x5bdb41 (0xe984 bytes) 000006[0] 1787734229.146696 read 0x1fec00 thru 0x2002e1 (0x16e2 bytes) truncating to largest ever: 0x6077d3 000010[0] 1787734229.206142 trunc from 0x5bdb42 to 0x6077d2 (0x49c91 bytes) 000015[0] 1787734229.289926 mapread 0x3a0aa9 thru 0x3a662e (0x5b86 bytes) 000016[0] 1787734229.369601 write 0x4eb4b2 thru 0x4f9c79 (0xe7c8 bytes) 000019[1] 1787734229.421794 read 0x1a859f thru 0x1b38cd (0xb32f bytes) 000020[1] 1787734229.443965 read 0x39a317 thru 0x39d1d8 (0x2ec2 bytes) 000021[0] 1787734229.562650 mapread 0x549a46 thru 0x550f60 (0x751b bytes) 000022[1] 1787734229.605717 mapwrite 0x37062f thru 0x376391 (0x5d63 bytes) truncating to largest ever: 0x8260d8 000026[0] 1787734229.877986 trunc from 0x6077d3 to 0x8260d7 (0x21e905 bytes) 000028[1] 1787734229.942088 read 0x766514 thru 0x77541c (0xef09 bytes) 000032[1] 1787734230.060193 read 0x131152 thru 0x1345f9 (0x34a8 bytes) 000033[0] 1787734230.075207 mapread 0x6036b7 thru 0x60894a (0x5294 bytes) 000034[0] 1787734230.093264 mapwrite 0x3d1bea thru 0x3d30f9 (0x1510 bytes) truncating to largest ever: 0x8362ca 000036[1] 1787734230.276922 trunc from 0x8260d8 to 0x8362c9 (0x101f2 bytes) 000038[0] 1787734230.385777 trunc from 0x8362ca to 0x0e88dd (0x74d9ec bytes) 000039[0] 1787734230.473933 trunc from 0x0e88de to 0x3c74c1 (0x2debe4 bytes) 000042[1] 1787734230.532004 write 0x91d7dd thru 0x91f9fa (0x221e bytes) 000043[1] 1787734230.563905 read 0x7e408a thru 0x7f187e (0xd7f5 bytes) 000044[1] 1787734230.585061 read 0x6b7c33 thru 0x6c4efe (0xd2cc bytes) 000045[1] 1787734230.593746 mapwrite 0x2db251 thru 0x2e71aa (0xbf5a bytes) 000051[0] 1787734230.706652 write 0x6940ea thru 0x69889c (0x47b3 bytes) 000052[1] 1787734230.763330 mapwrite 0x2656fd thru 0x2735ac (0xdeb0 bytes) 000053[1] 1787734230.859007 write 0x4b9896 thru 0x4c3566 (0x9cd1 bytes) 000059[1] 1787734230.974509 read 0x27f506 thru 0x289b1a (0xa615 bytes) 000060[1] 1787734230.984126 mapwrite 0x4d3f3f thru 0x4ddb09 (0x9bcb bytes) 000067[0] 1787734231.272084 trunc from 0x91f9fb to 0x16d154 (0x7b28a6 bytes) 000068[1] 1787734231.383443 mapwrite 0x74724a thru 0x747e1c (0xbd3 bytes) 000070[1] 1787734231.571008 mapread 0x3c6d92 thru 0x3d3ecd (0xd13c bytes) 000073[0] 1787734231.623267 write 0x968591 thru 0x9729a1 (0xa411 bytes) 000074[1] 1787734231.685138 read 0x11c5ce thru 0x12bb97 (0xf5ca bytes) 000076[1] 1787734231.768057 trunc from 0x9729a2 to 0x05b871 (0x917130 bytes) 000077[0] 1787734231.833567 read 0x04d065 thru 0x0597a0 (0xc73c bytes) 000078[0] 1787734231.879673 mapread 0x01818b thru 0x02735e (0xf1d4 bytes) 000080[1] 1787734231.937912 read 0x059d66 thru 0x05b871 (0x1b0c bytes) 000083[1] 1787734232.007367 mapwrite 0x95b0e8 thru 0x960a19 (0x5932 bytes) 000084[1] 1787734232.324659 mapwrite 0x29e8ea thru 0x2ac5a4 (0xdcbb bytes) 000086[0] 1787734232.414157 write 0x0004a0 thru 0x005a12 (0x5573 bytes) 000089[0] 1787734232.492465 mapread 0x295232 thru 0x2a3397 (0xe166 bytes) 000091[0] 1787734232.590840 read 0x490e23 thru 0x4913b6 (0x594 bytes) 000096[1] 1787734232.663768 trunc from 0x960a1a to 0x495da6 (0x4cac73 bytes) 000097[0] 1787734232.766643 read 0x1745e3 thru 0x183048 (0xea66 bytes) 000098[1] 1787734232.816631 mapwrite 0x37db0e thru 0x37db9d (0x90 bytes) 000099[0] 1787734232.931192 trunc from 0x495da7 to 0x3633bd (0x1329e9 bytes) 000100[1] 1787734232.977433 trunc from 0x3633be to 0x765c48 (0x40288b bytes) 000101[0] 1787734233.066184 read 0x312823 thru 0x312ab2 (0x290 bytes) 000102[1] 1787734233.106136 read 0x575243 thru 0x582e85 (0xdc43 bytes) 000103[1] 1787734233.153339 trunc from 0x765c49 to 0x38c094 (0x3d9bb4 bytes) 000106[1] 1787734233.234617 read 0x165f19 thru 0x166128 (0x210 bytes) 000107[0] 1787734233.309935 mapwrite 0x9ab725 thru 0x9b0f7b (0x5857 bytes) 000108[0] 1787734233.558131 mapread 0x88ad65 thru 0x88c2ce (0x156a bytes) 000110[0] 1787734233.656304 read 0x623c98 thru 0x6296f8 (0x5a61 bytes) 000112[1] 1787734233.685572 read 0x94fc73 thru 0x95f7db (0xfb69 bytes) 000114[1] 1787734233.726174 read 0x69f602 thru 0x6a4522 (0x4f21 bytes) 000116[0] 1787734233.752355 read 0x144d5f thru 0x151c7d (0xcf1f bytes) 000119[0] 1787734233.767489 write 0x03a35f thru 0x04120a (0x6eac bytes) 000121[1] 1787734233.811630 write 0x20df48 thru 0x211585 (0x363e bytes) 000122[0] 1787734233.879883 write 0x681172 thru 0x682025 (0xeb4 bytes) 000123[1] 1787734233.911480 mapread 0x48f12e thru 0x4993c8 (0xa29b bytes) 000124[0] 1787734233.965903 mapread 0x22d5f6 thru 0x236905 (0x9310 bytes) 000130[0] 1787734234.027609 mapread 0x229874 thru 0x229abc (0x249 bytes) 000131[0] 1787734234.028446 mapread 0x0fafd5 thru 0x0ff28a (0x42b6 bytes) 000132[1] 1787734234.045610 mapwrite 0x06b401 thru 0x06fc1d (0x481d bytes) 000133[0] 1787734234.133116 read 0x43071f thru 0x43635e (0x5c40 bytes) 000134[1] 1787734234.145732 write 0x6b76b4 thru 0x6b8940 (0x128d bytes) 000136[1] 1787734234.191536 read 0x47cf25 thru 0x47d593 (0x66f bytes) 000138[1] 1787734234.197423 mapwrite 0x3418bc thru 0x34dbf3 (0xc338 bytes) 000141[0] 1787734234.276944 read 0x244e8b thru 0x254a84 (0xfbfa bytes) 000143[0] 1787734234.316173 mapwrite 0x6f30cc thru 0x6febe4 (0xbb19 bytes) 000145[0] 1787734234.427114 mapwrite 0x638734 thru 0x63c673 (0x3f40 bytes) 000147[0] 1787734234.492714 read 0x26a65d thru 0x2775d8 (0xcf7c bytes) 000148[1] 1787734234.520203 mapread 0x5be504 thru 0x5ca5bf (0xc0bc bytes) 000153[0] 1787734234.576293 read 0x618bf3 thru 0x624473 (0xb881 bytes) 000155[0] 1787734234.587193 trunc from 0x9b0f7c to 0x550507 (0x460a74 bytes) 000159[1] 1787734234.676789 mapread 0x435bb0 thru 0x43709f (0x14f0 bytes) 000164[0] 1787734234.775740 mapwrite 0x4db17f thru 0x4df68c (0x450e bytes) truncating to largest ever: 0x862333 000166[1] 1787734234.968805 trunc from 0x550508 to 0x862332 (0x311e2b bytes) 000168[0] 1787734235.061033 trunc from 0x862333 to 0x0ce9b0 (0x793982 bytes) 000169[0] 1787734235.096134 trunc from 0x0ce9b1 to 0x067a89 (0x66f27 bytes) 000171[0] 1787734235.142025 mapread 0x0168c8 thru 0x017310 (0xa49 bytes) 000175[1] 1787734235.223490 read 0x01e879 thru 0x027f31 (0x96b9 bytes) 000176[0] 1787734235.251563 read 0x01df79 thru 0x02831e (0xa3a6 bytes) 000177[1] 1787734235.274301 trunc from 0x067a8a to 0x4ff536 (0x497aad bytes) 000178[1] 1787734235.350426 mapread 0x2c1da5 thru 0x2c8efd (0x7159 bytes) 000183[0] 1787734235.476684 trunc from 0x4ff537 to 0x03772a (0x4c7e0c bytes) 000184[1] 1787734235.548499 mapwrite 0x84cb03 thru 0x858137 (0xb635 bytes) 000186[0] 1787734235.716427 write 0x635fe4 thru 0x63b4ed (0x550a bytes) 000187[1] 1787734235.751157 trunc from 0x858138 to 0x13517d (0x722fba bytes) 000188[1] 1787734235.790657 read 0x0ab167 thru 0x0b60b9 (0xaf53 bytes) 000189[0] 1787734235.831626 read 0x03ede3 thru 0x0415db (0x27f9 bytes) 000190[0] 1787734235.875957 trunc from 0x13517e to 0x3f86e9 (0x2c356c bytes) 000191[0] 1787734235.923311 mapread 0x314a11 thru 0x321bd5 (0xd1c5 bytes) 000198[0] 1787734236.028188 mapwrite 0x51ad90 thru 0x51fe74 (0x50e5 bytes) 000199[0] 1787734236.173592 read 0x2f4f74 thru 0x2f5e22 (0xeaf bytes) 000200[0] 1787734236.179843 trunc from 0x51fe75 to 0x6f27dd (0x1d2969 bytes) 000201[0] 1787734236.212493 mapwrite 0x726021 thru 0x734bc0 (0xeba0 bytes) 000202[0] 1787734236.362060 mapread 0x70c7da thru 0x7197b1 (0xcfd8 bytes) 000203[1] 1787734236.383905 read 0x40b04e thru 0x414cf8 (0x9cab bytes) 000207[1] 1787734236.437618 mapread 0x50a07d thru 0x5103e6 (0x636a bytes) 000208[1] 1787734236.460977 read 0x54459f thru 0x550b6a (0xc5cc bytes) 000209[1] 1787734236.474292 mapread 0x1678f1 thru 0x175649 (0xdd59 bytes) 000210[1] 1787734236.581974 trunc from 0x734bc1 to 0x4b7602 (0x27d5be bytes) 000213[1] 1787734236.727284 write 0x66ffe1 thru 0x67d83f (0xd85f bytes) 000217[1] 1787734236.772025 mapwrite 0x782048 thru 0x7900e3 (0xe09c bytes) 000219[0] 1787734236.986734 mapwrite 0x4a2165 thru 0x4b20bb (0xff57 bytes) 000221[1] 1787734237.212049 read 0x707c19 thru 0x709d0d (0x20f5 bytes) 000222[1] 1787734237.265153 write 0x9587f5 thru 0x966a26 (0xe232 bytes) 000224[0] 1787734237.320061 mapread 0x6b02b9 thru 0x6bca25 (0xc76d bytes) 000225[0] 1787734237.388893 trunc from 0x966a27 to 0x205bcb (0x760e5b bytes) 000227[0] 1787734237.523457 trunc from 0x205bcc to 0x013f44 (0x1f1c87 bytes) 000229[1] 1787734237.543420 trunc from 0x013f45 to 0x0f38d5 (0xdf991 bytes) 000230[0] 1787734237.601498 trunc from 0x0f38d6 to 0x6d14c5 (0x5ddbf0 bytes) 000234[0] 1787734237.653421 trunc from 0x6d14c6 to 0x407e4b (0x2c967a bytes) 000236[0] 1787734237.726894 trunc from 0x407e4c to 0x253290 (0x1b4bbb bytes) 000241[1] 1787734237.777679 read 0x0c686b thru 0x0d5ba3 (0xf339 bytes) 000244[1] 1787734237.817149 mapread 0x1e2f6c thru 0x1e5277 (0x230c bytes) 000245[0] 1787734237.825749 trunc from 0x253291 to 0x246906 (0xc98a bytes) 000246[0] 1787734237.869960 read 0x082581 thru 0x08eedb (0xc95b bytes) 000247[1] 1787734237.907901 trunc from 0x246907 to 0x19a92d (0xabfd9 bytes) 000248[0] 1787734237.962675 mapwrite 0x5516b2 thru 0x554e69 (0x37b8 bytes) 000249[0] 1787734238.181664 mapread 0x438f68 thru 0x439a49 (0xae2 bytes) 000254[1] 1787734238.243323 mapwrite 0x1821bc thru 0x18d76c (0xb5b1 bytes) 000255[1] 1787734238.440073 read 0x3e234c thru 0x3e9abd (0x7772 bytes) 000256[0] 1787734238.459945 read 0x1e33b0 thru 0x1e5024 (0x1c75 bytes) 000262[1] 1787734238.540132 mapread 0x1d01d5 thru 0x1d2707 (0x2533 bytes) 000263[0] 1787734238.562175 write 0x68087c thru 0x68cc6a (0xc3ef bytes) 000266[1] 1787734238.601672 trunc from 0x68cc6b to 0x1dd901 (0x4af369 bytes) 000267[0] 1787734238.687137 read 0x09e96c thru 0x0ae5e3 (0xfc78 bytes) 000271[1] 1787734238.746001 mapread 0x19f18c thru 0x19ffb9 (0xe2e bytes) 000273[1] 1787734238.782291 mapwrite 0x27138f thru 0x272b73 (0x17e5 bytes) 000276[0] 1787734238.949997 mapwrite 0x4ac4d3 thru 0x4adb6d (0x169b bytes) 000277[1] 1787734239.072885 trunc from 0x4adb6e to 0x5b4f3f (0x1073d2 bytes) truncating to largest ever: 0x9ef98f 000279[1] 1787734239.145499 trunc from 0x5b4f40 to 0x9ef98e (0x43aa4f bytes) 000285[1] 1787734239.252201 trunc from 0x9ef98f to 0x0b4b70 (0x93ae1e bytes) 000286[1] 1787734239.274989 mapread 0x033a38 thru 0x041776 (0xdd3f bytes) 000287[1] 1787734239.337761 read 0x060020 thru 0x0675b7 (0x7598 bytes) 000292[0] 1787734239.369934 mapread 0x0452f7 thru 0x049cbf (0x49c9 bytes) 000294[1] 1787734239.445750 mapwrite 0x197c31 thru 0x19b075 (0x3445 bytes) 000295[1] 1787734239.565804 write 0x101724 thru 0x1099ee (0x82cb bytes) 000297[1] 1787734239.596350 mapread 0x15e825 thru 0x160110 (0x18ec bytes) 000299[1] 1787734239.679554 write 0x0a7d55 thru 0x0ae485 (0x6731 bytes) 000302[0] 1787734239.704823 mapread 0x1158de thru 0x117de5 (0x2508 bytes) 000303[1] 1787734239.765678 read 0x135e8f thru 0x14544e (0xf5c0 bytes) 000304[1] 1787734239.799790 write 0x218390 thru 0x218d56 (0x9c7 bytes) 000306[0] 1787734239.868595 read 0x217568 thru 0x218d56 (0x17ef bytes) 000307[1] 1787734239.937988 mapwrite 0x405e45 thru 0x40c765 (0x6921 bytes) 000308[1] 1787734240.131149 read 0x1650d8 thru 0x174034 (0xef5d bytes) 000314[1] 1787734240.258654 write 0x1e5027 thru 0x1efd48 (0xad22 bytes) 000315[1] 1787734240.260416 mapwrite 0x38c5a4 thru 0x399eb2 (0xd90f bytes) 000316[0] 1787734240.348521 mapread 0x2c419f thru 0x2c97db (0x563d bytes) 000317[0] 1787734240.429998 mapread 0x2994ba thru 0x2a82ed (0xee34 bytes) 000319[0] 1787734240.471605 write 0x56bac7 thru 0x576fb0 (0xb4ea bytes) 000320[0] 1787734240.496457 mapwrite 0x57bdea thru 0x57dcfd (0x1f14 bytes) 000323[1] 1787734240.768112 read 0x422368 thru 0x42d0f4 (0xad8d bytes) 000324[0] 1787734240.815121 read 0x3f8ea6 thru 0x406892 (0xd9ed bytes) 000327[0] 1787734240.836263 mapwrite 0x20d99c thru 0x21b497 (0xdafc bytes) 000333[1] 1787734241.104697 mapread 0x3f45bf thru 0x3f655c (0x1f9e bytes) 000335[0] 1787734241.142024 mapwrite 0x40a5f2 thru 0x40fb9e (0x55ad bytes) 000336[0] 1787734241.370979 mapread 0x44f7f2 thru 0x4541c8 (0x49d7 bytes) 000340[0] 1787734241.455009 trunc from 0x57dcfe to 0x9011ef (0x3834f2 bytes) 000344[1] 1787734241.605408 mapwrite 0x8c836c thru 0x8d6892 (0xe527 bytes) 000345[1] 1787734241.770603 trunc from 0x9011f0 to 0x622b86 (0x2de669 bytes) 000350[0] 1787734241.880148 trunc from 0x622b87 to 0x85ab85 (0x237fff bytes) 000351[0] 1787734241.941584 write 0x4ce58b thru 0x4d7fae (0x9a24 bytes) 000352[1] 1787734241.993766 mapread 0x0ed398 thru 0x0f0324 (0x2f8d bytes) 000353[1] 1787734242.085649 mapwrite 0x08e6fa thru 0x090265 (0x1b6c bytes) 000354[1] 1787734242.187384 trunc from 0x85ab86 to 0x8a0289 (0x45704 bytes) 000356[0] 1787734242.301912 trunc from 0x8a028a to 0x03987e (0x866a0b bytes) 000357[0] 1787734242.444545 read 0x00d335 thru 0x00ddb8 (0xa84 bytes) 000358[1] 1787734242.523187 mapwrite 0x57e94a thru 0x58a542 (0xbbf9 bytes) 000360[0] 1787734242.761273 read 0x540866 thru 0x54585c (0x4ff7 bytes) 000361[0] 1787734242.798134 mapwrite 0x148ee4 thru 0x1517fd (0x891a bytes) 000362[0] 1787734242.952616 trunc from 0x58a543 to 0x5b7082 (0x2cb40 bytes) 000366[0] 1787734243.066977 mapread 0x01f10d thru 0x0283ec (0x92e0 bytes) 000367[1] 1787734243.095644 write 0x306c8b thru 0x3119e4 (0xad5a bytes) 000368[0] 1787734243.163698 trunc from 0x5b7083 to 0x21db15 (0x39956d bytes) 000369[1] 1787734243.224742 trunc from 0x21db16 to 0x466488 (0x248973 bytes) 000372[0] 1787734243.336096 mapread 0x2e7c58 thru 0x2f42e2 (0xc68b bytes) 000373[0] 1787734243.420630 mapwrite 0x78649d thru 0x78bf59 (0x5abd bytes) 000374[0] 1787734243.549203 read 0x057982 thru 0x05e864 (0x6ee3 bytes) 000376[0] 1787734243.637700 mapwrite 0x20486b thru 0x205ef0 (0x1686 bytes) 000377[0] 1787734243.759808 mapwrite 0x7ddf7e thru 0x7eb1f6 (0xd279 bytes) 000380[0] 1787734243.970711 mapread 0x65dfc7 thru 0x662b25 (0x4b5f bytes) 000383[1] 1787734244.022003 mapwrite 0x38e634 thru 0x391112 (0x2adf bytes) 000385[1] 1787734244.189159 mapread 0x4837b9 thru 0x48e1e2 (0xaa2a bytes) 000389[1] 1787734244.281810 mapwrite 0x3b0aa9 thru 0x3b2023 (0x157b bytes) 000390[0] 1787734244.477698 write 0x52c794 thru 0x539c41 (0xd4ae bytes) 000392[1] 1787734244.571385 trunc from 0x7eb1f7 to 0x7a8481 (0x42d75 bytes) 000396[1] 1787734244.685979 mapwrite 0x9d2410 thru 0x9dae51 (0x8a42 bytes) 000400[0] 1787734244.917523 trunc from 0x9dae52 to 0x1b0c90 (0x82a1c1 bytes) 000401[1] 1787734244.987352 mapwrite 0x719e60 thru 0x722414 (0x85b5 bytes) 000402[0] 1787734245.149400 mapwrite 0x965aff thru 0x9732a9 (0xd7ab bytes) 000403[0] 1787734245.415951 mapwrite 0x390c2a thru 0x39f37f (0xe756 bytes) 000404[1] 1787734245.609666 write 0x0737a7 thru 0x077d96 (0x45f0 bytes) 000407[0] 1787734245.720389 write 0x662daf thru 0x66affc (0x824e bytes) 000411[0] 1787734245.851620 read 0x899931 thru 0x8a4e78 (0xb548 bytes) 000414[1] 1787734245.991642 trunc from 0x9732aa to 0x4584a8 (0x51ae01 bytes) 000416[1] 1787734246.125351 write 0x9191bc thru 0x91d5b1 (0x43f6 bytes) 000417[1] 1787734246.156455 write 0x497c30 thru 0x49a18b (0x255c bytes) 000419[0] 1787734246.208577 mapread 0x702177 thru 0x70de55 (0xbcdf bytes) 000420[0] 1787734246.373182 write 0x7ceded thru 0x7dcd3d (0xdf51 bytes) 000422[1] 1787734246.413205 trunc from 0x91d5b2 to 0x61365a (0x309f57 bytes) 000423[0] 1787734246.453794 read 0x23b715 thru 0x23bbfd (0x4e9 bytes) 000424[1] 1787734246.469559 mapwrite 0x455c09 thru 0x460bb7 (0xafaf bytes) 000425[0] 1787734246.584956 mapread 0x09e23d thru 0x0a46a3 (0x6467 bytes) 000426[0] 1787734246.622688 mapread 0x1a0e9c thru 0x1a5862 (0x49c7 bytes) 000427[0] 1787734246.682612 trunc from 0x61365b to 0x667628 (0x53fce bytes) 000432[1] 1787734246.990871 mapread 0x05a48a thru 0x064f27 (0xaa9e bytes) 000435[1] 1787734247.114725 mapwrite 0x92813e thru 0x92e829 (0x66ec bytes) 000436[0] 1787734247.409392 read 0x2cd09b thru 0x2d46ab (0x7611 bytes) 000437[0] 1787734247.455094 trunc from 0x92e82a to 0x84cf79 (0xe18b0 bytes) 000438[1] 1787734247.519096 mapread 0x31cf28 thru 0x32333f (0x6418 bytes) 000440[1] 1787734247.619502 mapread 0x78c0f6 thru 0x795b98 (0x9aa3 bytes) 000443[0] 1787734247.680556 mapread 0x15d1b3 thru 0x16b52e (0xe37c bytes) 000446[1] 1787734247.705667 write 0x019f8b thru 0x01edb0 (0x4e26 bytes) 000451[1] 1787734247.773344 read 0x361a07 thru 0x368cbe (0x72b8 bytes) 000452[1] 1787734247.788604 mapwrite 0x0d6098 thru 0x0dcb40 (0x6aa9 bytes) 000453[0] 1787734247.899800 mapwrite 0x294b94 thru 0x295ad9 (0xf46 bytes) 000454[1] 1787734248.006058 mapread 0x132ee6 thru 0x1335bc (0x6d7 bytes) 000455[1] 1787734248.030376 trunc from 0x84cf7a to 0x1e7c24 (0x665355 bytes) 000457[1] 1787734248.072186 mapread 0x061d68 thru 0x06610a (0x43a3 bytes) 000458[1] 1787734248.110463 write 0x134fa5 thru 0x1446a2 (0xf6fe bytes) 000459[1] 1787734248.136121 trunc from 0x1e7c25 to 0x98c072 (0x7a444e bytes) 000462[1] 1787734248.224206 read 0x715a49 thru 0x7231b0 (0xd768 bytes) 000463[0] 1787734248.260467 write 0x4fb7ed thru 0x4ffc19 (0x442d bytes) 000464[1] 1787734248.301500 write 0x477457 thru 0x47bcfa (0x48a4 bytes) 000468[1] 1787734248.486587 trunc from 0x98c073 to 0x081bcf (0x90a4a3 bytes) 000469[0] 1787734248.574996 read 0x017179 thru 0x018420 (0x12a8 bytes) 000471[0] 1787734248.636837 mapread 0x02f4c3 thru 0x03a6a0 (0xb1de bytes) 000473[0] 1787734248.703647 trunc from 0x081bd0 to 0x09c26b (0x1a69c bytes) 000474[1] 1787734248.751402 write 0x253477 thru 0x263331 (0xfebb bytes) 000475[1] 1787734248.793403 mapread 0x1d9038 thru 0x1de0bc (0x5085 bytes) 000478[1] 1787734248.874928 trunc from 0x263332 to 0x60082b (0x39d4fa bytes) 000481[0] 1787734249.016619 read 0x2c3d04 thru 0x2cc952 (0x8c4f bytes) 000485[0] 1787734249.170833 mapwrite 0x1dd9be thru 0x1ea78d (0xcdd0 bytes) 000489[1] 1787734249.582365 trunc from 0x60082c to 0x4edc8a (0x112ba1 bytes) 000490[0] 1787734249.654121 write 0x2a6eec thru 0x2a7199 (0x2ae bytes) 000491[1] 1787734249.732199 write 0x51f1c9 thru 0x525120 (0x5f58 bytes) 000492[0] 1787734249.773864 mapread 0x0ed29f thru 0x0f4d36 (0x7a98 bytes) 000495[1] 1787734249.931079 read 0x47341a thru 0x481833 (0xe41a bytes) 000496[0] 1787734249.963060 mapwrite 0x90dea7 thru 0x919810 (0xb96a bytes) 000497[0] 1787734250.150734 write 0x2768b3 thru 0x27bb26 (0x5274 bytes) 000498[1] 1787734250.213402 read 0x4dd32e thru 0x4def6d (0x1c40 bytes) 000499[0] 1787734250.315237 mapwrite 0x013320 thru 0x022678 (0xf359 bytes) 000500[0] 1787734250.412900 mapwrite 0x797c90 thru 0x7a7615 (0xf986 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4250 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x7b521 000003[0] 1787734251.565708 trunc from 00000000 to 0x07b520 (0x7b521 bytes) truncating to largest ever: 0x18c3bd 000004[1] 1787734251.605253 trunc from 0x07b521 to 0x18c3bc (0x110e9c bytes) 000005[0] 1787734251.728345 write 0x01f000 thru 0x027c5a (0x8c5b bytes) 000006[0] 1787734251.779602 mapread 0x08d000 thru 0x091e62 (0x4e63 bytes) 000007[0] 1787734251.886736 read 0x06f000 thru 0x07c7d5 (0xd7d6 bytes) 000008[0] 1787734251.943189 write 0x0bb000 thru 0x0c8f64 (0xdf65 bytes) 000009[1] 1787734251.955338 mapwrite 0x967000 thru 0x974e75 (0xde76 bytes) 000010[1] 1787734252.205266 write_OD 0x401000 thru 0x402fff (0x2000 bytes) 000011[1] 1787734252.356965 write_OD 0x9c8000 thru 0x9d3fff (0xc000 bytes) 000013[1] 1787734252.536934 mapwrite 0x4ce000 thru 0x4cebb8 (0xbb9 bytes) 000014[0] 1787734252.746369 mapwrite 0x7ed000 thru 0x7f209f (0x50a0 bytes) truncating to largest ever: 0x48cb3e 000016[0] 1787734253.082680 trunc from 0x9d4000 to 0x48cb3d (0x5474c2 bytes) 000021[0] 1787734253.167945 trunc from 0x48cb3e to 0x467606 (0x25537 bytes) truncating to largest ever: 0x4bd6e0 000023[1] 1787734253.227725 trunc from 0x467607 to 0x4bd6df (0x560d9 bytes) truncating to largest ever: 0x9dbd8f 000024[0] 1787734253.257570 trunc from 0x4bd6e0 to 0x9dbd8e (0x51e6af bytes) 000026[0] 1787734253.368639 write 0x3ef000 thru 0x3fa2eb (0xb2ec bytes) 000028[1] 1787734253.407544 read_OD 0x999000 thru 0x99dfff (0x5000 bytes) 000029[1] 1787734253.496040 read_OD 0x087000 thru 0x08ffff (0x9000 bytes) 000030[1] 1787734253.513868 read_OD 0x976000 thru 0x982fff (0xd000 bytes) 000031[1] 1787734253.523248 mapread 0x6df000 thru 0x6ec029 (0xd02a bytes) 000032[1] 1787734253.586794 trunc from 0x9dbd8f to 0x14e587 (0x88d807 bytes) 000034[1] 1787734253.664576 mapwrite 0x60d000 thru 0x616466 (0x9467 bytes) 000036[1] 1787734253.987899 mapwrite 0x361000 thru 0x36d759 (0xc75a bytes) 000037[0] 1787734254.223777 read 0x4b0000 thru 0x4bf20d (0xf20e bytes) truncating to largest ever: 0x9ddfed 000038[0] 1787734254.273376 trunc from 0x616467 to 0x9ddfec (0x3c7b86 bytes) 000040[0] 1787734254.406682 mapread 0x3b7000 thru 0x3bcf39 (0x5f3a bytes) 000043[0] 1787734254.486187 trunc from 0x9ddfed to 0x73322d (0x2aadbf bytes) 000045[0] 1787734254.598644 mapread 0x5ed000 thru 0x5edeb9 (0xeba bytes) 000049[1] 1787734254.647145 mapread 0x5a4000 thru 0x5aba55 (0x7a56 bytes) 000050[1] 1787734254.750339 trunc from 0x73322e to 0x4236c4 (0x30fb69 bytes) 000052[0] 1787734254.906464 trunc from 0x4236c5 to 0x6c78c1 (0x2a41fd bytes) 000053[1] 1787734254.965560 mapread 0x5ad000 thru 0x5b5917 (0x8918 bytes) 000054[1] 1787734255.036880 mapwrite 0x2bc000 thru 0x2c8ff4 (0xcff5 bytes) 000055[1] 1787734255.244768 mapread 0x623000 thru 0x6285fc (0x55fd bytes) 000056[0] 1787734255.340131 trunc from 0x6c78c2 to 0x8e56b1 (0x21ddf0 bytes) 000058[1] 1787734255.462383 mapread 0x7a2000 thru 0x7ac5ab (0xa5ac bytes) 000059[1] 1787734255.567610 mapwrite 0x652000 thru 0x65adeb (0x8dec bytes) 000062[0] 1787734255.819016 mapwrite 0x87f000 thru 0x88bdc9 (0xcdca bytes) 000065[1] 1787734256.090196 mapwrite 0x155000 thru 0x15c815 (0x7816 bytes) 000066[1] 1787734256.226498 read_OD 0x6d9000 thru 0x6e1fff (0x9000 bytes) 000067[1] 1787734256.246136 mapread 0x878000 thru 0x87b50a (0x350b bytes) 000074[1] 1787734256.293502 mapwrite 0x478000 thru 0x47b3ce (0x33cf bytes) 000075[0] 1787734256.454849 write 0x58a000 thru 0x58c567 (0x2568 bytes) 000076[1] 1787734256.521802 read_OD 0x397000 thru 0x3a2fff (0xc000 bytes) 000080[1] 1787734256.559077 mapread 0x602000 thru 0x60731e (0x531f bytes) 000081[1] 1787734256.580034 write_OD 0x6db000 thru 0x6e6fff (0xc000 bytes) 000084[1] 1787734256.665558 mapread 0x002000 thru 0x010fb9 (0xefba bytes) 000085[0] 1787734256.701466 trunc from 0x8e56b2 to 0x6ba225 (0x22b48c bytes) 000086[0] 1787734256.794340 mapwrite 0x001000 thru 0x00f882 (0xe883 bytes) 000087[1] 1787734256.914169 trunc from 0x6ba226 to 0x136213 (0x584012 bytes) 000091[0] 1787734257.003915 trunc from 0x136214 to 0x3814cc (0x24b2b9 bytes) 000093[1] 1787734257.083900 read_OD 0x32a000 thru 0x32afff (0x1000 bytes) 000094[0] 1787734257.113614 trunc from 0x3814cd to 0x3958fe (0x14432 bytes) 000095[1] 1787734257.168254 trunc from 0x3958ff to 0x25831b (0x13d5e3 bytes) 000096[0] 1787734257.232629 read 0x046000 thru 0x052666 (0xc667 bytes) 000098[0] 1787734257.360040 mapread 0x199000 thru 0x199dc1 (0xdc2 bytes) 000099[1] 1787734257.361510 read_OD 0x141000 thru 0x149fff (0x9000 bytes) 000100[1] 1787734257.371517 read_OD 0x0c1000 thru 0x0c2fff (0x2000 bytes) 000105[1] 1787734257.379093 mapwrite 0x912000 thru 0x9157e6 (0x37e7 bytes) 000106[0] 1787734257.820533 mapwrite 0x88e000 thru 0x89cd6b (0xed6c bytes) 000107[0] 1787734258.305558 mapread 0x39b000 thru 0x39ffad (0x4fae bytes) 000108[0] 1787734258.340888 mapread 0x66e000 thru 0x67825c (0xa25d bytes) 000109[0] 1787734258.346681 read 0x82d000 thru 0x832eea (0x5eeb bytes) 000110[0] 1787734258.388201 mapwrite 0x4b5000 thru 0x4c0833 (0xb834 bytes) 000112[1] 1787734258.723599 mapwrite 0x10c000 thru 0x10fa6c (0x3a6d bytes) 000115[1] 1787734259.019820 mapwrite 0x0d9000 thru 0x0d968d (0x68e bytes) 000117[0] 1787734259.217405 mapwrite 0x844000 thru 0x847795 (0x3796 bytes) 000118[1] 1787734259.550469 write_OD 0x74a000 thru 0x74ffff (0x6000 bytes) 000121[0] 1787734259.685630 mapread 0x367000 thru 0x368701 (0x1702 bytes) 000125[0] 1787734259.715081 mapread 0x8ca000 thru 0x8d4d28 (0xad29 bytes) 000127[1] 1787734259.779680 mapread 0x2ec000 thru 0x2ee552 (0x2553 bytes) 000128[1] 1787734259.824288 mapwrite 0x55d000 thru 0x56a99a (0xd99b bytes) 000129[1] 1787734260.127516 trunc from 0x9157e7 to 0x3ac710 (0x5690d6 bytes) 000132[1] 1787734260.224430 mapread 0x1db000 thru 0x1e9d18 (0xed19 bytes) 000133[1] 1787734260.406383 mapread 0x097000 thru 0x099d37 (0x2d38 bytes) 000137[0] 1787734260.559731 trunc from 0x3ac711 to 0x738c0c (0x38c4fc bytes) 000139[1] 1787734260.714282 read_OD 0x3b8000 thru 0x3bdfff (0x6000 bytes) 000142[1] 1787734260.763127 mapread 0x547000 thru 0x556114 (0xf115 bytes) 000144[1] 1787734260.807621 read_OD 0x479000 thru 0x483fff (0xb000 bytes) 000146[1] 1787734260.838392 mapread 0x2c2000 thru 0x2c4345 (0x2346 bytes) 000150[0] 1787734260.882283 mapread 0x126000 thru 0x128581 (0x2582 bytes) 000151[0] 1787734260.937926 read 0x1a0000 thru 0x1ae922 (0xe923 bytes) 000152[0] 1787734260.952320 trunc from 0x738c0d to 0x98e730 (0x255b24 bytes) 000153[1] 1787734261.014539 write_OD 0x68e000 thru 0x68ffff (0x2000 bytes) 000156[0] 1787734261.130572 read 0x3ae000 thru 0x3b7a7e (0x9a7f bytes) 000158[1] 1787734261.147776 write_OD 0x97f000 thru 0x989fff (0xb000 bytes) 000159[1] 1787734261.306062 write_OD 0x662000 thru 0x66cfff (0xb000 bytes) 000164[1] 1787734261.454706 trunc from 0x98e731 to 0x58e313 (0x40041d bytes) 000166[1] 1787734261.607592 mapread 0x090000 thru 0x09dadd (0xdade bytes) 000169[1] 1787734261.699119 trunc from 0x58e314 to 0x334538 (0x259ddb bytes) 000171[1] 1787734261.837176 write_OD 0x721000 thru 0x727fff (0x7000 bytes) 000172[1] 1787734262.045358 read_OD 0x3a4000 thru 0x3a7fff (0x4000 bytes) 000173[1] 1787734262.068846 write_OD 0x8f6000 thru 0x8fbfff (0x6000 bytes) 000175[1] 1787734262.259753 write_OD 0x45b000 thru 0x45cfff (0x2000 bytes) 000177[0] 1787734262.385971 mapread 0x7a5000 thru 0x7aac9e (0x5c9f bytes) 000178[0] 1787734262.437059 write 0x76c000 thru 0x76fe49 (0x3e4a bytes) 000180[0] 1787734262.459066 write 0x88b000 thru 0x88f808 (0x4809 bytes) 000181[0] 1787734262.486789 trunc from 0x8fc000 to 0x0bfdc9 (0x83c236 bytes) 000182[1] 1787734262.538801 read_OD 0x011000 thru 0x01efff (0xe000 bytes) 000183[0] 1787734262.583051 write 0x852000 thru 0x85571f (0x3720 bytes) 000185[1] 1787734262.602411 mapwrite 0x02d000 thru 0x03b15b (0xe15c bytes) 000186[1] 1787734262.898066 trunc from 0x855720 to 0x371b14 (0x4e3c0b bytes) 000192[0] 1787734262.986929 write 0x407000 thru 0x4078d3 (0x8d4 bytes) 000193[1] 1787734263.034046 write_OD 0x6de000 thru 0x6e1fff (0x4000 bytes) 000195[1] 1787734263.144145 trunc from 0x6e2000 to 0x0be50e (0x623af1 bytes) 000196[0] 1787734263.265046 trunc from 0x0be50f to 0x91e2c1 (0x85fdb3 bytes) 000197[0] 1787734263.318347 read 0x44e000 thru 0x45520f (0x7210 bytes) 000202[0] 1787734263.379977 trunc from 0x91e2c2 to 0x888d02 (0x955bf bytes) 000203[0] 1787734263.458353 mapread 0x659000 thru 0x65f880 (0x6881 bytes) 000204[0] 1787734263.529866 read 0x587000 thru 0x596f02 (0xff03 bytes) 000207[0] 1787734263.644921 mapread 0x63c000 thru 0x642812 (0x6813 bytes) 000208[0] 1787734263.658383 write 0x171000 thru 0x179d51 (0x8d52 bytes) 000209[1] 1787734263.687404 trunc from 0x888d03 to 0x438223 (0x450adf bytes) 000213[1] 1787734263.766989 mapread 0x2cc000 thru 0x2d1fa4 (0x5fa5 bytes) 000217[1] 1787734263.855499 read_OD 0x0cf000 thru 0x0d2fff (0x4000 bytes) 000219[0] 1787734263.877065 write 0x624000 thru 0x631d6b (0xdd6c bytes) 000220[0] 1787734263.890774 write 0x8e8000 thru 0x8f0cde (0x8cdf bytes) 000221[0] 1787734263.893390 mapread 0x5bf000 thru 0x5c9d3a (0xad3b bytes) 000223[1] 1787734263.917683 mapwrite 0x4e2000 thru 0x4eb55e (0x955f bytes) 000225[0] 1787734264.018969 write 0x57d000 thru 0x58aa3d (0xda3e bytes) 000228[0] 1787734264.070395 trunc from 0x8f0cdf to 0x353d12 (0x59cfcc bytes) 000229[1] 1787734264.095447 trunc from 0x353d13 to 0x66eef2 (0x31b1e0 bytes) 000230[1] 1787734264.120109 trunc from 0x66eef3 to 0x32a2ec (0x344c06 bytes) 000231[1] 1787734264.143532 write_OD 0x4cc000 thru 0x4d2fff (0x7000 bytes) 000232[0] 1787734264.186697 mapread 0x10c000 thru 0x117014 (0xb015 bytes) 000234[1] 1787734264.221400 mapwrite 0x2aa000 thru 0x2ac24a (0x224b bytes) 000235[1] 1787734264.301278 trunc from 0x4d3000 to 0x63d107 (0x16a108 bytes) 000236[0] 1787734264.322860 trunc from 0x63d108 to 0x8cd149 (0x290042 bytes) 000238[1] 1787734264.369833 read_OD 0x2e2000 thru 0x2e3fff (0x2000 bytes) 000239[1] 1787734264.386950 read_OD 0x560000 thru 0x569fff (0xa000 bytes) 000242[0] 1787734264.400381 trunc from 0x8cd14a to 0x066e6a (0x8662df bytes) 000243[1] 1787734264.427014 mapwrite 0x5b9000 thru 0x5c840a (0xf40b bytes) 000245[1] 1787734264.511204 trunc from 0x5c840b to 0x9dbdf3 (0x4139e9 bytes) 000250[0] 1787734264.560103 mapwrite 0x6b0000 thru 0x6bd357 (0xd358 bytes) 000255[1] 1787734264.659528 trunc from 0x9dbdf4 to 0x74dc7e (0x28e175 bytes) 000257[1] 1787734264.707006 read_OD 0x353000 thru 0x357fff (0x5000 bytes) 000259[0] 1787734264.725355 trunc from 0x74dc7f to 0x7eb5c3 (0x9d945 bytes) 000261[0] 1787734264.749690 read 0x760000 thru 0x76a2f9 (0xa2fa bytes) 000265[1] 1787734264.785971 trunc from 0x7eb5c4 to 0x55b691 (0x28ff32 bytes) 000266[1] 1787734264.816947 mapwrite 0x1b3000 thru 0x1bf0ed (0xc0ee bytes) 000267[1] 1787734264.894490 mapread 0x421000 thru 0x424392 (0x3393 bytes) 000269[1] 1787734264.915467 trunc from 0x55b692 to 0x1516a2 (0x409fef bytes) 000271[0] 1787734264.932174 trunc from 0x1516a3 to 0x7d5ec8 (0x684826 bytes) 000272[1] 1787734264.984744 mapwrite 0x3ae000 thru 0x3bc12c (0xe12d bytes) 000273[1] 1787734265.087326 read_OD 0x291000 thru 0x298fff (0x8000 bytes) 000274[1] 1787734265.103655 write_OD 0x788000 thru 0x792fff (0xb000 bytes) 000275[1] 1787734265.145469 mapwrite 0x60d000 thru 0x611c9a (0x4c9b bytes) 000276[0] 1787734265.227797 write 0x23b000 thru 0x2459a4 (0xa9a5 bytes) 000278[1] 1787734265.267200 mapread 0x3bc000 thru 0x3bed9f (0x2da0 bytes) 000282[1] 1787734265.324334 mapread 0x3f7000 thru 0x4056a0 (0xe6a1 bytes) 000285[1] 1787734265.361776 write_OD 0x0f1000 thru 0x0f7fff (0x7000 bytes) 000287[0] 1787734265.453842 mapread 0x5e2000 thru 0x5ea439 (0x843a bytes) 000288[0] 1787734265.485572 trunc from 0x7d5ec9 to 0x098949 (0x73d57f bytes) 000292[0] 1787734265.538518 mapwrite 0x9d0000 thru 0x9d9d5d (0x9d5e bytes) 000299[1] 1787734265.799803 mapwrite 0x895000 thru 0x8a10be (0xc0bf bytes) 000301[0] 1787734266.020154 trunc from 0x9d9d5e to 0x0be078 (0x91bce5 bytes) 000302[0] 1787734266.072870 write 0x62b000 thru 0x62e408 (0x3409 bytes) 000305[0] 1787734266.157333 mapwrite 0x9f0000 thru 0x9f5e2d (0x5e2e bytes) 000306[0] 1787734266.340957 read 0x671000 thru 0x67b271 (0xa272 bytes) 000307[1] 1787734266.373296 mapread 0x38e000 thru 0x396bf6 (0x8bf7 bytes) 000311[0] 1787734266.408845 write 0x396000 thru 0x396764 (0x765 bytes) 000316[0] 1787734266.475802 write 0x24a000 thru 0x24ada4 (0xda5 bytes) 000317[0] 1787734266.481018 trunc from 0x9f5e2e to 0x41615e (0x5dfccf bytes) 000320[0] 1787734266.552822 read 0x1b3000 thru 0x1b471c (0x171d bytes) 000322[1] 1787734266.581257 trunc from 0x41615f to 0x89b12f (0x484fd1 bytes) 000323[0] 1787734266.638773 trunc from 0x89b130 to 0x422b8b (0x4785a4 bytes) 000326[0] 1787734266.674011 mapread 0x11d000 thru 0x11f99e (0x299f bytes) 000327[1] 1787734266.691671 trunc from 0x422b8c to 0x629611 (0x206a86 bytes) 000328[1] 1787734266.745556 write_OD 0x290000 thru 0x297fff (0x8000 bytes) 000329[1] 1787734266.804561 mapread 0x57a000 thru 0x581fff (0x8000 bytes) 000332[0] 1787734266.842186 mapread 0x0e5000 thru 0x0e7496 (0x2497 bytes) 000334[1] 1787734266.865440 read_OD 0x50f000 thru 0x51afff (0xc000 bytes) 000336[1] 1787734266.877951 mapread 0x3cc000 thru 0x3d1ff6 (0x5ff7 bytes) 000339[0] 1787734266.902350 mapwrite 0x639000 thru 0x63dd12 (0x4d13 bytes) 000341[0] 1787734267.042553 write 0x7e5000 thru 0x7eedbe (0x9dbf bytes) 000343[1] 1787734267.052583 read_OD 0x2b4000 thru 0x2bafff (0x7000 bytes) 000344[1] 1787734267.125440 trunc from 0x7eedbf to 0x9b6663 (0x1c78a5 bytes) 000346[0] 1787734267.207484 read 0x053000 thru 0x061e53 (0xee54 bytes) 000347[0] 1787734267.245489 trunc from 0x9b6664 to 0x518b02 (0x49db61 bytes) 000348[0] 1787734267.309822 read 0x48d000 thru 0x4908d7 (0x38d8 bytes) 000349[0] 1787734267.342361 trunc from 0x518b03 to 0x3736bc (0x1a5446 bytes) 000353[0] 1787734267.392253 trunc from 0x3736bd to 0x77b4b8 (0x407dfc bytes) 000358[0] 1787734267.479830 read 0x608000 thru 0x617abb (0xfabc bytes) 000360[1] 1787734267.502843 mapread 0x580000 thru 0x58ddea (0xddeb bytes) 000364[1] 1787734267.549467 mapwrite 0x9e2000 thru 0x9e877e (0x677f bytes) 000365[0] 1787734267.661716 read 0x839000 thru 0x83afc1 (0x1fc2 bytes) 000367[1] 1787734267.695539 trunc from 0x9e877f to 0x68168f (0x3670ef bytes) 000371[1] 1787734267.796508 trunc from 0x681690 to 0x670c45 (0x10a4a bytes) 000372[0] 1787734267.814434 mapread 0x0dd000 thru 0x0dfd8d (0x2d8e bytes) 000373[0] 1787734267.850956 mapread 0x210000 thru 0x215fb8 (0x5fb9 bytes) 000375[1] 1787734267.880015 write_OD 0x579000 thru 0x587fff (0xf000 bytes) 000377[1] 1787734267.948687 mapwrite 0x2f1000 thru 0x2faee3 (0x9ee4 bytes) 000378[0] 1787734268.058827 mapread 0x5b4000 thru 0x5bbb5a (0x7b5b bytes) 000382[0] 1787734268.113621 read 0x0bb000 thru 0x0c6907 (0xb908 bytes) 000383[0] 1787734268.121411 mapread 0x46c000 thru 0x47bb11 (0xfb12 bytes) 000384[1] 1787734268.151494 mapwrite 0x909000 thru 0x91321a (0xa21b bytes) 000386[0] 1787734268.322581 read 0x07c000 thru 0x08b884 (0xf885 bytes) 000387[1] 1787734268.354638 trunc from 0x91321b to 0x98d8af (0x7a695 bytes) 000388[1] 1787734268.448293 write_OD 0x0be000 thru 0x0c7fff (0xa000 bytes) 000389[0] 1787734268.559892 write 0x6ce000 thru 0x6d0ddc (0x2ddd bytes) 000390[0] 1787734268.609514 mapread 0x84b000 thru 0x852e75 (0x7e76 bytes) 000392[1] 1787734268.644943 trunc from 0x98d8b0 to 0x5333b0 (0x45a4ff bytes) 000394[0] 1787734268.722176 write 0x802000 thru 0x8054cd (0x34ce bytes) 000395[1] 1787734268.764768 trunc from 0x8054ce to 0x6333a8 (0x1d2125 bytes) 000400[1] 1787734268.894936 mapwrite 0x9d0000 thru 0x9df46e (0xf46f bytes) 000408[0] 1787734269.103152 mapwrite 0x1c7000 thru 0x1c8a47 (0x1a48 bytes) 000409[1] 1787734269.235485 trunc from 0x9df46f to 0x0e4a9c (0x8fa9d2 bytes) 000411[1] 1787734269.309998 read_OD 0x012000 thru 0x018fff (0x7000 bytes) 000414[1] 1787734269.330383 mapread 0x084000 thru 0x08c2c4 (0x82c5 bytes) 000415[1] 1787734269.346947 trunc from 0x0e4a9d to 0x45a5a7 (0x375b0b bytes) 000416[0] 1787734269.391379 read 0x095000 thru 0x097f01 (0x2f02 bytes) 000420[0] 1787734269.419392 mapwrite 0x5b6000 thru 0x5bfd0a (0x9d0b bytes) 000421[0] 1787734269.584803 trunc from 0x5bfd0b to 0x71864e (0x158944 bytes) 000422[0] 1787734269.610942 read 0x619000 thru 0x61cedf (0x3ee0 bytes) 000428[1] 1787734269.660927 write_OD 0x5e0000 thru 0x5e5fff (0x6000 bytes) 000432[1] 1787734269.720325 mapread 0x717000 thru 0x718143 (0x1144 bytes) 000435[0] 1787734269.832438 write 0x397000 thru 0x39bfd7 (0x4fd8 bytes) 000438[0] 1787734269.900167 write 0x14a000 thru 0x159d06 (0xfd07 bytes) 000441[0] 1787734269.968179 trunc from 0x71864f to 0x23388c (0x4e4dc2 bytes) 000442[1] 1787734269.987810 write_OD 0x626000 thru 0x630fff (0xb000 bytes) 000443[0] 1787734270.051844 read 0x234000 thru 0x23d106 (0x9107 bytes) 000447[0] 1787734270.101135 trunc from 0x631000 to 0x3cd29f (0x263d60 bytes) 000448[1] 1787734270.146682 trunc from 0x3cd2a0 to 0x6c8004 (0x2fad65 bytes) 000449[1] 1787734270.177702 read_OD 0x47f000 thru 0x48afff (0xc000 bytes) 000450[0] 1787734270.201938 trunc from 0x6c8005 to 0x660443 (0x67bc1 bytes) 000454[1] 1787734270.259406 trunc from 0x660444 to 0x7577e1 (0xf739e bytes) 000455[1] 1787734270.285527 trunc from 0x7577e2 to 0x67ea23 (0xd8dbe bytes) 000459[0] 1787734270.342892 mapwrite 0x9f2000 thru 0x9fb1c9 (0x91ca bytes) 000460[0] 1787734270.580323 mapread 0x18d000 thru 0x191ac0 (0x4ac1 bytes) 000461[0] 1787734270.615958 mapread 0x29f000 thru 0x2a3161 (0x4162 bytes) 000464[0] 1787734270.678778 read 0x461000 thru 0x463d96 (0x2d97 bytes) 000466[0] 1787734270.691581 write 0x9f7000 thru 0x9f8678 (0x1679 bytes) 000467[1] 1787734270.710801 mapwrite 0x28d000 thru 0x29a8f7 (0xd8f8 bytes) 000469[0] 1787734270.872910 mapwrite 0x023000 thru 0x024d91 (0x1d92 bytes) 000471[1] 1787734270.952974 mapread 0x982000 thru 0x98d6a1 (0xb6a2 bytes) 000472[1] 1787734271.046149 read_OD 0x7a2000 thru 0x7adfff (0xc000 bytes) 000474[0] 1787734271.086676 read 0x72a000 thru 0x737bdc (0xdbdd bytes) 000476[1] 1787734271.127733 read_OD 0x04e000 thru 0x04efff (0x1000 bytes) 000478[0] 1787734271.145387 mapread 0x6d2000 thru 0x6e0689 (0xe68a bytes) 000479[0] 1787734271.203701 mapwrite 0x84b000 thru 0x84f478 (0x4479 bytes) 000481[0] 1787734271.375245 read 0x404000 thru 0x406d20 (0x2d21 bytes) 000482[0] 1787734271.405383 mapread 0x5df000 thru 0x5e2685 (0x3686 bytes) 000483[1] 1787734271.444980 read_OD 0x817000 thru 0x81bfff (0x5000 bytes) 000485[0] 1787734271.502745 trunc from 0x9fb1ca to 0x404ded (0x5f63dc bytes) 000486[0] 1787734271.555456 write 0x07a000 thru 0x08411b (0xa11c bytes) 000490[1] 1787734271.564175 mapwrite 0x9b2000 thru 0x9c1812 (0xf813 bytes) 000493[1] 1787734271.793026 write_OD 0x2ff000 thru 0x303fff (0x5000 bytes) 000495[1] 1787734271.939647 trunc from 0x9c1813 to 0x5f7979 (0x3c9e99 bytes) 000499[0] 1787734272.034760 write 0x645000 thru 0x64af17 (0x5f18 bytes) 000500[0] 1787734272.050797 read 0x2a0000 thru 0x2a7933 (0x7934 bytes) All operations completed A-OK! PASS 16a (53s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 04:51:17 (1787734277) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.445791 s, 18.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.306757 s, 27.3 MB/s Chance of close/open is 1 in 50 Seed set to 4280 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000003[0] 1787734281.122447 mapwrite 0x85e69e thru 0x85fffe (0x1961 bytes) 000004[1] 1787734281.287342 mapread 0x708c56 thru 0x7098e4 (0xc8f bytes) 000006[1] 1787734281.393917 mapread 0x170e83 thru 0x170f30 (0xae bytes) 000007[0] 1787734281.443997 mapread 0x825f0f thru 0x825f22 (0x14 bytes) 000008[0] 1787734281.464658 mapwrite 0x3ba7f5 thru 0x3bc2fa (0x1b06 bytes) truncating to largest ever: 0x1d6c7d 000009[0] 1787734281.614272 trunc from 0x85ffff to 0x1d6c7c (0x689382 bytes) 000010[0] 1787734281.729013 mapwrite 0x45f3d4 thru 0x460740 (0x136d bytes) 000016[1] 1787734282.006197 mapwrite 0x93dd2a thru 0x93ed24 (0xffb bytes) 000017[0] 1787734282.128905 read 0x69ab13 thru 0x69c2ba (0x17a8 bytes) 000019[0] 1787734282.179860 write 0x75914a thru 0x759495 (0x34c bytes) 000021[0] 1787734282.236445 write 0x574163 thru 0x5753e2 (0x1280 bytes) 000023[1] 1787734282.312450 read 0x58e356 thru 0x58e9c7 (0x672 bytes) 000027[0] 1787734282.375988 mapread 0x7ee3b8 thru 0x7ef4cc (0x1115 bytes) 000029[0] 1787734282.426165 read 0x3d770d thru 0x3d8f5d (0x1851 bytes) 000030[0] 1787734282.444599 trunc from 0x93ed25 to 0x10ab2e (0x8341f6 bytes) 000031[1] 1787734282.492599 mapwrite 0x3a9143 thru 0x3aa68c (0x154a bytes) truncating to largest ever: 0x943466 000032[0] 1787734282.585464 trunc from 0x3aa68d to 0x943465 (0x598dd9 bytes) 000033[1] 1787734282.641261 trunc from 0x943466 to 0x409d4d (0x539718 bytes) 000037[1] 1787734282.721062 mapwrite 0x8fde52 thru 0x8ff8fe (0x1aad bytes) 000038[0] 1787734282.831046 read 0x03ed87 thru 0x03fe56 (0x10d0 bytes) truncating to largest ever: 0x96fefb 000039[1] 1787734282.868002 trunc from 0x8ff8ff to 0x96fefa (0x705fc bytes) 000042[1] 1787734282.944432 mapread 0x831dd1 thru 0x8324b4 (0x6e4 bytes) 000044[0] 1787734282.982197 mapwrite 0x7a7d04 thru 0x7a87ab (0xaa8 bytes) 000046[0] 1787734283.120585 read 0x40b357 thru 0x40bc95 (0x93f bytes) 000048[0] 1787734283.156804 write 0x297319 thru 0x29774e (0x436 bytes) 000052[1] 1787734283.234923 read 0x29ca2f thru 0x29cb61 (0x133 bytes) 000053[1] 1787734283.261459 mapread 0x26765c thru 0x267aa7 (0x44c bytes) 000055[0] 1787734283.302583 write 0x4395be thru 0x439ee6 (0x929 bytes) 000056[1] 1787734283.333120 mapwrite 0x6cb924 thru 0x6cb9d1 (0xae bytes) 000057[1] 1787734283.486332 mapread 0x08c2dd thru 0x08d1d1 (0xef5 bytes) 000058[1] 1787734283.521194 read 0x0fdedd thru 0x0fee4f (0xf73 bytes) 000059[1] 1787734283.531344 mapread 0x19ceb7 thru 0x19d289 (0x3d3 bytes) 000061[0] 1787734283.598812 trunc from 0x96fefb to 0x631e02 (0x33e0f8 bytes) 000062[1] 1787734283.693708 write 0x17f7a8 thru 0x17fa8c (0x2e5 bytes) 000064[1] 1787734283.744952 mapread 0x007592 thru 0x00844b (0xeba bytes) 000065[0] 1787734283.785048 write 0x69c739 thru 0x69e353 (0x1c1b bytes) 000066[1] 1787734283.821109 trunc from 0x69e354 to 0x0c6c20 (0x5d7733 bytes) 000067[1] 1787734283.948118 mapwrite 0x917d35 thru 0x91964a (0x1916 bytes) 000068[1] 1787734284.052532 mapread 0x0aecc2 thru 0x0b0b20 (0x1e5f bytes) 000070[1] 1787734284.070658 read 0x2ae629 thru 0x2afb5d (0x1535 bytes) 000071[1] 1787734284.081088 mapwrite 0x8177cc thru 0x8182a8 (0xadd bytes) 000074[1] 1787734284.164661 write 0x01718c thru 0x017d0d (0xb82 bytes) 000075[0] 1787734284.170912 trunc from 0x91964b to 0x0e7633 (0x832017 bytes) 000081[0] 1787734284.305033 write 0x69892c thru 0x699422 (0xaf7 bytes) 000082[1] 1787734284.310587 trunc from 0x699423 to 0x630368 (0x690ba bytes) 000083[0] 1787734284.375649 trunc from 0x630369 to 0x7f5ef8 (0x1c5b90 bytes) truncating to largest ever: 0x9a80ab 000084[0] 1787734284.456582 trunc from 0x7f5ef9 to 0x9a80aa (0x1b21b2 bytes) 000086[0] 1787734284.530689 mapread 0x5adf74 thru 0x5aedd9 (0xe66 bytes) 000087[1] 1787734284.550443 mapwrite 0x94279e thru 0x942e27 (0x68a bytes) 000088[0] 1787734284.695281 write 0x551188 thru 0x552853 (0x16cc bytes) 000092[1] 1787734284.783291 mapread 0x49434c thru 0x4953a9 (0x105e bytes) 000097[1] 1787734284.844800 mapread 0x8bbd56 thru 0x8bd388 (0x1633 bytes) 000099[1] 1787734284.902517 read 0x7ff80b thru 0x800091 (0x887 bytes) 000100[0] 1787734284.936051 trunc from 0x9a80ab to 0x1a1eee (0x8061bc bytes) 000102[0] 1787734285.010190 mapwrite 0x513082 thru 0x514661 (0x15e0 bytes) 000103[0] 1787734285.116029 write 0x6ce511 thru 0x6cfaf0 (0x15e0 bytes) 000104[0] 1787734285.123030 trunc from 0x6cfaf1 to 0x8df7a1 (0x20fcb1 bytes) 000105[0] 1787734285.150179 read 0x358d61 thru 0x359236 (0x4d6 bytes) 000106[0] 1787734285.157952 write 0x658666 thru 0x659c9b (0x1636 bytes) 000107[1] 1787734285.203905 read 0x45ff2b thru 0x4615a0 (0x1676 bytes) 000111[1] 1787734285.306329 read 0x4b188e thru 0x4b22a4 (0xa17 bytes) 000112[1] 1787734285.317834 mapread 0x18c17c thru 0x18dd8b (0x1c10 bytes) 000113[0] 1787734285.367830 write 0x1db418 thru 0x1dc611 (0x11fa bytes) 000115[0] 1787734285.409256 mapwrite 0x4757b2 thru 0x476029 (0x878 bytes) 000116[1] 1787734285.512711 mapread 0x377079 thru 0x378218 (0x11a0 bytes) 000118[0] 1787734285.574041 write 0x8f111d thru 0x8f1ce6 (0xbca bytes) 000120[0] 1787734285.653206 trunc from 0x8f1ce7 to 0x57616b (0x37bb7b bytes) 000121[0] 1787734285.682640 mapwrite 0x3e7c42 thru 0x3e8046 (0x405 bytes) 000122[1] 1787734285.841362 mapwrite 0x1d60ad thru 0x1d6e44 (0xd98 bytes) 000125[1] 1787734286.049907 mapread 0x531e9f thru 0x53261e (0x780 bytes) 000126[1] 1787734286.079539 read 0x21b8dd thru 0x21c148 (0x86c bytes) 000129[1] 1787734286.087027 write 0x818240 thru 0x819869 (0x162a bytes) 000132[1] 1787734286.122030 mapread 0x4323cd thru 0x432550 (0x184 bytes) 000135[0] 1787734286.153074 mapwrite 0x303fd2 thru 0x30528c (0x12bb bytes) 000136[1] 1787734286.283394 mapread 0x478649 thru 0x478f3d (0x8f5 bytes) 000137[0] 1787734286.297636 write 0x87da27 thru 0x87ef75 (0x154f bytes) truncating to largest ever: 0x9efda7 000138[1] 1787734286.312810 trunc from 0x87ef76 to 0x9efda6 (0x170e31 bytes) 000142[1] 1787734286.445277 trunc from 0x9efda7 to 0x2dbfea (0x713dbc bytes) 000143[0] 1787734286.476184 mapread 0x0b9879 thru 0x0bac60 (0x13e8 bytes) 000144[1] 1787734286.541323 mapwrite 0x18ba24 thru 0x18c4d3 (0xab0 bytes) 000145[0] 1787734286.664234 mapwrite 0x756881 thru 0x756f48 (0x6c8 bytes) 000147[0] 1787734286.853496 mapread 0x48a617 thru 0x48bbf5 (0x15df bytes) 000149[0] 1787734286.921967 mapwrite 0x1f8c95 thru 0x1fa24d (0x15b9 bytes) 000150[1] 1787734287.039785 write 0x5a92d8 thru 0x5a93d6 (0xff bytes) 000151[1] 1787734287.086509 mapwrite 0x1eb71d thru 0x1ebb3f (0x423 bytes) 000152[1] 1787734287.228973 trunc from 0x756f49 to 0x0f7131 (0x65fe17 bytes) 000155[1] 1787734287.304052 write 0x77be4e thru 0x77bff9 (0x1ac bytes) 000157[1] 1787734287.365258 mapread 0x72eb11 thru 0x7301a5 (0x1695 bytes) 000158[1] 1787734287.389027 read 0x4e1321 thru 0x4e169b (0x37b bytes) 000159[0] 1787734287.394941 mapwrite 0x96bf94 thru 0x96df68 (0x1fd5 bytes) 000161[0] 1787734287.501346 mapread 0x06aea9 thru 0x06c420 (0x1578 bytes) 000163[1] 1787734287.540058 mapwrite 0x74e5c7 thru 0x74ff64 (0x199e bytes) 000164[1] 1787734287.631302 trunc from 0x96df69 to 0x569cb8 (0x4042b0 bytes) 000165[0] 1787734287.659021 mapwrite 0x214b13 thru 0x21589c (0xd8a bytes) 000168[0] 1787734287.821783 write 0x7b5e6b thru 0x7b759e (0x1734 bytes) 000169[1] 1787734287.846979 mapread 0x24c7d5 thru 0x24dfc5 (0x17f1 bytes) 000171[1] 1787734287.900340 trunc from 0x7b759f to 0x62dda6 (0x1897f8 bytes) 000176[0] 1787734288.069759 trunc from 0x62dda7 to 0x23054e (0x3fd858 bytes) 000178[0] 1787734288.148002 read 0x142c68 thru 0x144a3f (0x1dd8 bytes) 000182[1] 1787734288.255527 mapwrite 0x33dc4e thru 0x33ea38 (0xdeb bytes) 000184[0] 1787734288.381886 write 0x0a11cf thru 0x0a1dba (0xbec bytes) 000185[1] 1787734288.461526 mapwrite 0x8c1afb thru 0x8c347b (0x1981 bytes) 000188[1] 1787734288.592275 mapwrite 0x046458 thru 0x046fe6 (0xb8f bytes) 000191[0] 1787734288.712683 mapread 0x5752b3 thru 0x575c22 (0x970 bytes) 000194[0] 1787734288.789932 mapwrite 0x8c2739 thru 0x8c3208 (0xad0 bytes) 000197[1] 1787734289.076238 trunc from 0x8c347c to 0x49a9e6 (0x428a95 bytes) 000199[1] 1787734289.163080 mapread 0x0725ac thru 0x073adc (0x1531 bytes) 000202[1] 1787734289.236816 mapwrite 0x20eb56 thru 0x2105e4 (0x1a8f bytes) 000204[1] 1787734289.358591 read 0x28e449 thru 0x29008f (0x1c47 bytes) 000207[0] 1787734289.404728 trunc from 0x49a9e7 to 0x56bc11 (0xd122b bytes) 000208[0] 1787734289.498813 trunc from 0x56bc12 to 0x8a59f2 (0x339de1 bytes) 000210[1] 1787734289.629027 trunc from 0x8a59f3 to 0x95b76d (0xb5d7b bytes) 000213[0] 1787734289.721387 trunc from 0x95b76e to 0x4b751e (0x4a424f bytes) 000217[1] 1787734289.918369 mapwrite 0x042ee5 thru 0x044c6c (0x1d88 bytes) 000219[0] 1787734290.068819 write 0x6fc732 thru 0x6fccf3 (0x5c2 bytes) 000220[0] 1787734290.086966 trunc from 0x6fccf4 to 0x043cd6 (0x6b901d bytes) 000222[0] 1787734290.125270 mapwrite 0x29452e thru 0x295886 (0x1359 bytes) 000223[1] 1787734290.211311 mapwrite 0x50607c thru 0x506eaf (0xe34 bytes) 000224[0] 1787734290.344608 mapwrite 0x7650a6 thru 0x7658ad (0x808 bytes) 000225[1] 1787734290.436024 mapread 0x261b4c thru 0x262958 (0xe0d bytes) 000227[1] 1787734290.488544 read 0x1e19bc thru 0x1e321a (0x185f bytes) 000228[1] 1787734290.494542 mapread 0x278fad thru 0x27a84d (0x18a1 bytes) 000233[1] 1787734290.559909 mapread 0x12f2f7 thru 0x12f82a (0x534 bytes) 000234[1] 1787734290.575656 mapread 0x33f13d thru 0x33f330 (0x1f4 bytes) 000235[0] 1787734290.589696 mapwrite 0x97964f thru 0x97b0e3 (0x1a95 bytes) 000240[0] 1787734290.783610 mapread 0x3b1af9 thru 0x3b2c5f (0x1167 bytes) 000244[0] 1787734290.822941 write 0x0559c8 thru 0x05671b (0xd54 bytes) 000249[0] 1787734290.916979 read 0x4a3f94 thru 0x4a4b86 (0xbf3 bytes) 000251[1] 1787734290.967624 mapread 0x8ae6b2 thru 0x8af46a (0xdb9 bytes) 000252[1] 1787734291.030457 read 0x064fe0 thru 0x0667cb (0x17ec bytes) 000253[1] 1787734291.044757 write 0x681a9a thru 0x6831a5 (0x170c bytes) 000255[0] 1787734291.097564 write 0x3ed57b thru 0x3ee522 (0xfa8 bytes) 000256[1] 1787734291.133708 read 0x91b2bc thru 0x91c724 (0x1469 bytes) 000257[1] 1787734291.169525 write 0x3bcac5 thru 0x3be914 (0x1e50 bytes) 000258[0] 1787734291.191327 mapwrite 0x50de3f thru 0x50f690 (0x1852 bytes) 000260[1] 1787734291.308943 write 0x3f255c thru 0x3f26c6 (0x16b bytes) 000261[0] 1787734291.358079 mapwrite 0x359595 thru 0x35aa7d (0x14e9 bytes) 000262[0] 1787734291.582825 trunc from 0x97b0e4 to 0x955329 (0x25dba bytes) 000263[0] 1787734291.604915 trunc from 0x95532a to 0x0624f7 (0x8f2e32 bytes) 000266[0] 1787734291.629307 mapwrite 0x0141f3 thru 0x0146f6 (0x504 bytes) 000268[0] 1787734291.739609 trunc from 0x0624f8 to 0x8bd986 (0x85b48f bytes) 000272[0] 1787734291.798027 write 0x1ac1ef thru 0x1ae099 (0x1eab bytes) 000274[0] 1787734291.830276 mapread 0x0fc477 thru 0x0fde89 (0x1a13 bytes) 000275[1] 1787734291.842503 read 0x41d83e thru 0x41e89c (0x105f bytes) 000276[0] 1787734291.860384 mapwrite 0x41d883 thru 0x41e3d5 (0xb53 bytes) 000277[1] 1787734291.926502 mapread 0x3c1b84 thru 0x3c229f (0x71c bytes) 000278[0] 1787734291.968980 mapwrite 0x690615 thru 0x690a31 (0x41d bytes) 000279[0] 1787734292.032010 read 0x6b9f4c thru 0x6bab3e (0xbf3 bytes) 000280[1] 1787734292.050780 mapwrite 0x1faaf8 thru 0x1fc9b0 (0x1eb9 bytes) 000282[0] 1787734292.214323 write 0x816f8d thru 0x8171ee (0x262 bytes) 000283[1] 1787734292.249714 mapwrite 0x54e1f6 thru 0x54f7ce (0x15d9 bytes) 000285[1] 1787734292.455622 read 0x0abf05 thru 0x0acf3b (0x1037 bytes) 000294[1] 1787734292.622998 mapread 0x311c79 thru 0x3126a2 (0xa2a bytes) 000295[0] 1787734292.633461 mapread 0x2a4139 thru 0x2a5280 (0x1148 bytes) 000297[1] 1787734292.679406 trunc from 0x8bd987 to 0x2d6d4c (0x5e6c3a bytes) 000298[0] 1787734292.725032 read 0x215dd7 thru 0x215e26 (0x50 bytes) 000299[0] 1787734292.765148 trunc from 0x2d6d4d to 0x6f28cf (0x41bb83 bytes) 000300[1] 1787734292.822063 mapwrite 0x09a77f thru 0x09b478 (0xcfa bytes) 000301[0] 1787734292.919011 read 0x465c1d thru 0x467326 (0x170a bytes) 000303[1] 1787734292.979549 mapread 0x272160 thru 0x272c1a (0xabb bytes) 000304[0] 1787734293.031295 mapread 0x4d60a5 thru 0x4d75df (0x153b bytes) 000305[1] 1787734293.066172 mapwrite 0x431a76 thru 0x431ad5 (0x60 bytes) 000308[1] 1787734293.170654 mapwrite 0x850821 thru 0x85195d (0x113d bytes) 000310[0] 1787734293.234750 mapwrite 0x81142b thru 0x811ede (0xab4 bytes) 000311[1] 1787734293.333282 mapread 0x58ed00 thru 0x58fa0e (0xd0f bytes) 000313[0] 1787734293.379563 mapwrite 0x95c5c1 thru 0x95d549 (0xf89 bytes) 000315[0] 1787734293.565835 mapread 0x0ecc9c thru 0x0ed5ac (0x911 bytes) 000316[1] 1787734293.623721 mapwrite 0x47afd9 thru 0x47c8bc (0x18e4 bytes) 000317[1] 1787734293.758173 trunc from 0x95d54a to 0x236f97 (0x7265b2 bytes) 000318[0] 1787734293.788532 mapwrite 0x62b311 thru 0x62c64d (0x133d bytes) 000320[1] 1787734293.886048 mapwrite 0x029251 thru 0x029fd1 (0xd81 bytes) 000324[0] 1787734294.091850 mapwrite 0x63f2f5 thru 0x63fd7a (0xa86 bytes) 000328[0] 1787734294.285989 read 0x4690ea thru 0x46a1ba (0x10d1 bytes) 000330[1] 1787734294.311975 trunc from 0x63fd7b to 0x43bc3a (0x204140 bytes) 000332[1] 1787734294.361038 mapwrite 0x17ba66 thru 0x17d021 (0x15bc bytes) 000333[0] 1787734294.467884 mapread 0x0b8de3 thru 0x0b9f06 (0x1124 bytes) 000334[1] 1787734294.520413 mapread 0x19d7f4 thru 0x19ea49 (0x1256 bytes) 000335[0] 1787734294.533087 read 0x38e262 thru 0x38fd4a (0x1ae9 bytes) 000341[1] 1787734294.589009 trunc from 0x43bc3b to 0x473504 (0x378ca bytes) 000342[0] 1787734294.646473 mapread 0x033bb9 thru 0x034764 (0xbac bytes) 000343[1] 1787734294.679862 write 0x400cce thru 0x401a74 (0xda7 bytes) 000344[0] 1787734294.688759 read 0x3c05d1 thru 0x3c07ab (0x1db bytes) 000345[0] 1787734294.790243 trunc from 0x473505 to 0x888ce0 (0x4157dc bytes) 000347[0] 1787734294.879965 mapread 0x67959f thru 0x67ab0d (0x156f bytes) 000348[1] 1787734294.980354 read 0x2fbada thru 0x2fc1f9 (0x720 bytes) 000349[0] 1787734295.011194 trunc from 0x888ce1 to 0x1ff0b8 (0x689c28 bytes) 000350[1] 1787734295.062143 trunc from 0x1ff0b9 to 0x4fcc75 (0x2fdbbd bytes) 000358[0] 1787734295.282274 write 0x6af180 thru 0x6b08f5 (0x1776 bytes) 000360[1] 1787734295.328879 mapwrite 0x79b4d2 thru 0x79d130 (0x1c5f bytes) 000361[1] 1787734295.439079 mapwrite 0x91211e thru 0x9140dc (0x1fbf bytes) 000362[0] 1787734295.607383 trunc from 0x9140dd to 0x269a79 (0x6aa663 bytes) 000363[1] 1787734295.683809 mapwrite 0x3fb974 thru 0x3fc290 (0x91d bytes) 000365[1] 1787734295.872664 mapread 0x1b8724 thru 0x1ba5eb (0x1ec8 bytes) 000368[0] 1787734295.906508 mapwrite 0x2a87a9 thru 0x2a9e58 (0x16b0 bytes) 000369[1] 1787734296.004037 write 0x2b4c57 thru 0x2b6306 (0x16b0 bytes) 000371[1] 1787734296.039292 mapwrite 0x45b85a thru 0x45bd37 (0x4de bytes) 000375[1] 1787734296.193711 mapread 0x41cb6d thru 0x41dac5 (0xf59 bytes) 000377[0] 1787734296.227318 trunc from 0x45bd38 to 0x910c92 (0x4b4f5b bytes) 000380[1] 1787734296.307297 mapread 0x0e2dd5 thru 0x0e481c (0x1a48 bytes) 000382[1] 1787734296.432950 mapread 0x1dca7e thru 0x1dda8d (0x1010 bytes) 000383[1] 1787734296.470708 write 0x98acd2 thru 0x98c987 (0x1cb6 bytes) 000384[1] 1787734296.501163 read 0x90729f thru 0x9085f5 (0x1357 bytes) 000385[1] 1787734296.523276 mapwrite 0x71e728 thru 0x7205c0 (0x1e99 bytes) 000387[0] 1787734296.683919 mapwrite 0x107a05 thru 0x1098e6 (0x1ee2 bytes) 000392[1] 1787734296.936890 trunc from 0x98c988 to 0x01b258 (0x97172f bytes) 000393[1] 1787734297.006382 write 0x751419 thru 0x75244d (0x1035 bytes) 000394[0] 1787734297.015044 write 0x6c068e thru 0x6c095b (0x2ce bytes) 000395[0] 1787734297.053457 mapwrite 0x863f0a thru 0x865cfe (0x1df5 bytes) 000396[1] 1787734297.200401 trunc from 0x865cff to 0x64e7b2 (0x21754c bytes) 000400[1] 1787734297.282551 read 0x2a2755 thru 0x2a3f5b (0x1807 bytes) 000401[1] 1787734297.287687 write 0x1b7cf3 thru 0x1b8dfa (0x1108 bytes) 000405[0] 1787734297.348044 trunc from 0x64e7b3 to 0x9c1152 (0x3729a0 bytes) 000406[0] 1787734297.451181 read 0x180da1 thru 0x182a60 (0x1cc0 bytes) 000407[1] 1787734297.498624 read 0x2ae071 thru 0x2ae274 (0x204 bytes) 000408[0] 1787734297.527774 mapread 0x62e3c9 thru 0x62f6fb (0x1333 bytes) 000409[1] 1787734297.557852 write 0x13110e thru 0x131e4d (0xd40 bytes) 000411[1] 1787734297.613316 write 0x39216f thru 0x393cf2 (0x1b84 bytes) 000414[0] 1787734297.674968 read 0x91fee6 thru 0x9216e0 (0x17fb bytes) 000418[0] 1787734297.759022 read 0x40194c thru 0x402f63 (0x1618 bytes) 000420[1] 1787734297.773035 trunc from 0x9c1153 to 0x9934bf (0x2dc93 bytes) 000423[1] 1787734297.894132 read 0x1d8873 thru 0x1d9167 (0x8f5 bytes) 000439[0] 1787734298.179986 write 0x939bf2 thru 0x93a8fc (0xd0b bytes) 000440[1] 1787734298.213990 mapread 0x5a5dca thru 0x5a661b (0x852 bytes) 000441[1] 1787734298.255058 mapread 0x7a7c3d thru 0x7a948d (0x1851 bytes) 000442[1] 1787734298.331822 mapwrite 0x2216c0 thru 0x22352c (0x1e6d bytes) 000444[0] 1787734298.475041 read 0x202257 thru 0x2031a3 (0xf4d bytes) 000445[1] 1787734298.503244 mapwrite 0x513488 thru 0x513676 (0x1ef bytes) 000447[0] 1787734298.608544 read 0x083901 thru 0x084d15 (0x1415 bytes) 000449[0] 1787734298.653283 write 0x48abf9 thru 0x48cb32 (0x1f3a bytes) 000451[0] 1787734298.700545 trunc from 0x9934c0 to 0x47104a (0x522475 bytes) 000454[0] 1787734298.850389 read 0x06b18f thru 0x06bfe4 (0xe56 bytes) 000456[0] 1787734298.915330 read 0x456448 thru 0x4566db (0x294 bytes) 000458[1] 1787734298.934426 mapwrite 0x179746 thru 0x179a31 (0x2ec bytes) 000459[1] 1787734299.065492 trunc from 0x47104b to 0x99a7df (0x529795 bytes) 000460[1] 1787734299.157912 mapwrite 0x2c185c thru 0x2c2e34 (0x15d9 bytes) 000462[0] 1787734299.277326 mapread 0x4b89ca thru 0x4b8cdf (0x316 bytes) 000463[1] 1787734299.356063 write 0x7e9d91 thru 0x7ea4eb (0x75b bytes) 000464[0] 1787734299.431628 trunc from 0x99a7e0 to 0x808c2a (0x191bb5 bytes) 000465[0] 1787734299.546369 write 0x8ec13d thru 0x8edef6 (0x1dba bytes) 000466[0] 1787734299.563140 mapread 0x5335bf thru 0x534271 (0xcb3 bytes) 000467[0] 1787734299.587004 mapwrite 0x4fa1ba thru 0x4fb9a8 (0x17ef bytes) 000469[0] 1787734299.673926 trunc from 0x8edef7 to 0x546b97 (0x3a735f bytes) 000472[0] 1787734299.829203 read 0x119236 thru 0x119e39 (0xc04 bytes) 000475[1] 1787734299.867777 read 0x4ef6e2 thru 0x4f002a (0x949 bytes) 000476[1] 1787734299.901571 trunc from 0x546b98 to 0x7013d1 (0x1ba83a bytes) 000477[0] 1787734299.951301 write 0x1751fa thru 0x1756dd (0x4e4 bytes) 000483[0] 1787734300.115952 trunc from 0x7013d2 to 0x9072ef (0x205f1e bytes) 000485[0] 1787734300.183552 mapread 0x38e380 thru 0x38f7b3 (0x1434 bytes) 000486[1] 1787734300.197491 mapwrite 0x06daaa thru 0x06e6da (0xc31 bytes) 000488[1] 1787734300.392982 mapread 0x37b13e thru 0x37b1b8 (0x7b bytes) 000491[1] 1787734300.467659 write 0x167316 thru 0x1676cb (0x3b6 bytes) 000494[0] 1787734300.512140 mapread 0x60bb00 thru 0x60c724 (0xc25 bytes) 000495[1] 1787734300.540192 read 0x4f2996 thru 0x4f452e (0x1b99 bytes) 000498[0] 1787734300.580818 mapwrite 0x694cdf thru 0x6961aa (0x14cc bytes) 000499[0] 1787734300.657689 write 0x856895 thru 0x857829 (0xf95 bytes) 000500[1] 1787734300.690627 write 0x73c201 thru 0x73c494 (0x294 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 04:51:46 (1787734306) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 04:51:50 (1787734310) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.336494 s, 24.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.304762 s, 27.5 MB/s ldlm.namespaces.MGC192.168.203.104@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff977c47b58800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff977c517f0000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff977c47b58800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff977c47b58800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3702 s, 10.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.2006 s, 32.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.4352 s, 43.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41841 s, 19.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.64503 s, 13.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.96789 s, 35.3 MB/s PASS 16d (43s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 04:52:33 (1787734353) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.249223 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138922 s, 7.5 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 04:52:39 (1787734359) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 04:53:07 (1787734387) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 04:53:34 (1787734414) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 04:53:40 (1787734420) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 04:53:46 (1787734426) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74313 s, 15.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.908733 s, 28.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.00323 s, 13.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.22884 s, 21.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62791 s, 16.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57241 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0165 s, 13.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.893997 s, 29.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21684 s, 21.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.89896 s, 9.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.97965 s, 8.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.24514 s, 11.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.50223 s, 10.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.33832 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.50505 s, 7.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.87363 s, 9.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07637 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.81238 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.45967 s, 5.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 04:54:23 (1787734463) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 All operations completed A-OK! All operations completed A-OK! Wed Aug 26 04:54:58 EDT 2026 + fsx 2 done: rc=0 Wed Aug 26 04:54:59 EDT 2026 + fsx 3 done: rc=0 Wed Aug 26 04:54:59 EDT 2026 + fsx 4 done: rc=0 Wed Aug 26 04:54:59 EDT 2026 + fsx 5 done: rc=0 Wed Aug 26 04:54:59 EDT 2026 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 04:55:06 (1787734506) fail_loc=0x8000030a PASS 17 (9s) == sanityn test 18: mmap sanity check =========================================================================================== 04:55:16 (1787734516) mmap test1: basic mmap operation (PASS, 0.12297s) mmap test2: MAP_PRIVATE not write back (PASS, 0.06471s) mmap test3: concurrent mmap ops on two nodes (PASS, 13.241s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3958s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.11239s) mmap test6: check mmap write/read content on two nodes (PASS, 0.20622s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3989s) mmap test11: multi-thread shared mmap (PASS, 1.2447s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1127s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 04:55:44 (1787734544) oleg304-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 04:55:48 (1787734548) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 04:55:55 (1787734555) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 04:56:01 (1787734561) now is 1787734563 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 new atime is 1787734624 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 04:57:09 (1787734629) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.3K 387 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 387 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 39.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234851 387 234464 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 387 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.3K 387 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234851 387 234464 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537417 387 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117266 402 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234851 387 234464 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 04:57:15 (1787734635) PASS 24b (5s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 04:57:20 (1787734640) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 04:57:27 (1787734647) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 04:57:30 (1787734650) utime: good mknod times 1787734650 <= 1787734651 <= 1787734651 for /mnt/lustre/f26a utime: good mknod times 1787734650 <= 1787734651 <= 1787734651 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 04:57:36 (1787734656) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 04:57:43 (1787734663) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-26 04:57:45.000000000 -0400 Birth: 2026-08-26 04:57:45.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-26 04:57:45.000000000 -0400 Birth: 2026-08-26 04:57:45.000000000 -0400 PASS 26c (8s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 04:57:53 (1787734673) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 04:58:03 (1787734683) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 04:58:10 (1787734690) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.104@tcp:/lustre 7542784 40960 7497728 1% /mnt/lustre2 PASS 31b (21s) == sanityn test 31r: open-rename(replace) race =========== 04:58:31 (1787734711) -rw-r--r-- 1 root root 0 Aug 26 04:58 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (8s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 04:58:40 (1787734720) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43132 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 04:58:46 (1787734726) PASS 31t (7s) == sanityn test 32b: lockless i/o ======================== 04:58:53 (1787734733) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (4s) == sanityn test 32c: contention detection ================ 04:58:57 (1787734737) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.17564 s, 44.6 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.52643 s, 34.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.88887 s, 18.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.14576 s, 16.7 MB/s contention_seconds 4, contention_events 22 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.61045 s, 20.1 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.90131 s, 18.1 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.70285 s, 30.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.76249 s, 29.7 MB/s PASS 32c (33s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 04:59:32 (1787734772) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 04:59:36 (1787734776) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 04:59:40 (1787734780) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 04:59:44 (1787734784) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 04:59:47 (1787734787) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid in IDLE state after 0 sec oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid in FULL state after 0 sec oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid in IDLE state after 0 sec oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid in FULL state after 0 sec oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff977c47b58800.ost_server_uuid in IDLE state after 0 sec oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff977c47b58800.ost_server_uuid in FULL state after 0 sec PASS 34 (133s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 05:02:00 (1787734920) fail_loc=0x80000317 Race attempt 0 Wait for 50452 50481 for 60 sec... - open/close 651 (time 1787734937.17 total 10.01 last 65.06) - open/close 1353 (time 1787734947.17 total 20.01 last 70.15) - open/close 2025 (time 1787734957.18 total 30.02 last 67.13) - open/close 2709 (time 1787734967.19 total 40.02 last 68.39) - open/close 3428 (time 1787734977.20 total 50.04 last 71.81) total: 4000 open/close in 58.04 seconds: 68.91 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.104@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.104@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 05:03:11 (1787734991) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg304-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3794 s, 38.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85748 s, 28.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7494656) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3912 s, 37.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7494656) after_dd(7438336) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.09376 s, 25.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7496704) after_dd(7441408) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.85737 s, 28.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.19678 s, 43.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7496704) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43461 s, 36.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7496704) after_dd(7443456) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.04165 s, 50.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7496704) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15548 s, 45.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47208 s, 35.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(9) *** before(7487488) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.06388 s, 49.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7492608) after_dd(7439360) after(7490560) PASS 36 (323s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 05:08:35 (1787735315) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 - create 1293 (time 1787735328.96 total 10.01 last 129.23) - create 2635 (time 1787735338.96 total 20.01 last 134.15) - create 4063 (time 1787735348.97 total 30.01 last 142.76) - create 5571 (time 1787735358.97 total 40.02 last 150.76) - create 6900 (time 1787735368.97 total 50.02 last 132.87) - create 8122 (time 1787735378.97 total 60.02 last 122.20) - create 9408 (time 1787735388.97 total 70.02 last 128.58) total: 10000 create in 75.07 seconds: 133.21 ops/second PASS 37 (85s) == sanityn test 39a: file mtime does not change after rename ========================================================== 05:10:01 (1787735401) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 05:10:08 (1787735408) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 05:10:16 (1787735416) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 05:10:24 (1787735424) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 05:10:33 (1787735433) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 05:10:51 (1787735451) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 05:11:07 (1787735467) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 05:11:22 (1787735482) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 05:11:37 (1787735497) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 05:11:51 (1787735511) open: File exists Conflict PASS 41b (13s) == sanityn test 41c: pdirops: create vs link ======================================================================== 05:12:04 (1787735524) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 05:12:18 (1787735538) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 05:12:33 (1787735553) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 05:12:45 (1787735565) Conflict PASS 41f (10s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 05:12:55 (1787735575) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 05:13:06 (1787735586) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 05:13:18 (1787735598) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1005s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 05:30:03 (1787736603) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 05:30:16 (1787736616) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 05:30:29 (1787736629) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 05:30:44 (1787736644) Conflict PASS 42d (13s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 05:30:57 (1787736657) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 05:31:10 (1787736670) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 05:31:24 (1787736684) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 05:31:38 (1787736698) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 05:31:53 (1787736713) PASS 43a (73s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 05:33:06 (1787736786) Conflict PASS 43b (12s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 05:33:18 (1787736798) Conflict PASS 43c (12s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 05:33:31 (1787736811) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (11s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 05:33:42 (1787736822) Conflict PASS 43e (12s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 05:33:55 (1787736835) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 05:34:06 (1787736846) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (12s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 05:34:18 (1787736858) Conflict PASS 43h (12s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 05:34:30 (1787736870) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 05:34:33 (1787736873) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (131s) == sanityn test 43k: unlink vs create ==================== 05:36:44 (1787737004) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1144s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 05:55:48 (1787738148) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Aug 26 05:55:56 EDT 2026 PASS 44a (12s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 05:56:01 (1787738161) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 05:56:13 (1787738173) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 05:56:27 (1787738187) Conflict PASS 44d (16s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 05:56:43 (1787738203) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 05:56:58 (1787738218) Conflict PASS 44f (12s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 05:57:10 (1787738230) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 05:57:23 (1787738243) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 05:57:37 (1787738257) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 05:57:41 (1787738261) PASS 45a (124s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 05:59:46 (1787738386) Conflict PASS 45b (12s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 05:59:58 (1787738398) Conflict PASS 45c (13s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 06:00:11 (1787738411) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 06:00:26 (1787738426) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 06:00:41 (1787738441) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (10s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 06:00:51 (1787738451) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (11s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 06:01:02 (1787738462) Conflict PASS 45h (12s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 06:01:14 (1787738474) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 06:01:17 (1787738477) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1185s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:21:02 (1787739662) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (12s) == sanityn test 46b: pdirops: link vs create ======================================================================== 06:21:15 (1787739675) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 06:21:28 (1787739688) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:21:41 (1787739701) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 06:21:54 (1787739714) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 06:22:07 (1787739727) Conflict PASS 46f (16s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 06:22:24 (1787739744) Conflict PASS 46g (15s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 06:22:40 (1787739760) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 06:22:54 (1787739774) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:22:57 (1787739777) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:23:02 (1787739782) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:23:05 (1787739785) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:23:09 (1787739789) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:23:12 (1787739792) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:23:16 (1787739796) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:23:20 (1787739800) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:23:23 (1787739803) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.180025 s, 56.9 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:23:35 (1787739815) 0+1 records in 0+1 records out 158 bytes copied, 0.0346287 s, 4.6 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218867) - No such process PASS 51a (8s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:23:44 (1787739824) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0132864 s, 77.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123913 s, 8.5 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:24:04 (1787739844) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0227982 s, 44.9 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:24:17 (1787739857) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607565 s, 17.3 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:24:25 (1787739865) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 06:24:35 (1787739875) llite.lustre-ffff977c47b58800.xattr_cache=1 llite.lustre-ffff977c517f0000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:37.000000000 -0400 Modify: 2026-08-26 06:24:37.000000000 -0400 Change: 2026-08-26 06:24:37.000000000 -0400 Birth: 2026-08-26 06:24:37.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:43.000000000 -0400 Modify: 2026-08-26 06:24:44.000000000 -0400 Change: 2026-08-26 06:24:44.000000000 -0400 Birth: 2026-08-26 06:24:44.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:50.000000000 -0400 Modify: 2026-08-26 06:24:50.000000000 -0400 Change: 2026-08-26 06:24:50.000000000 -0400 Birth: 2026-08-26 06:24:50.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:50.000000000 -0400 Modify: 2026-08-26 06:24:50.000000000 -0400 Change: 2026-08-26 06:24:50.000000000 -0400 Birth: 2026-08-26 06:24:50.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:56.000000000 -0400 Modify: 2026-08-26 06:24:57.000000000 -0400 Change: 2026-08-26 06:24:57.000000000 -0400 Birth: 2026-08-26 06:24:56.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 06:24:56.000000000 -0400 Modify: 2026-08-26 06:24:57.000000000 -0400 Change: 2026-08-26 06:24:57.000000000 -0400 Birth: 2026-08-26 06:24:56.000000000 -0400 PASS 54 (33s) == sanityn test 55a: rename vs unlink target dir ========= 06:25:08 (1787739908) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 06:25:23 (1787739923) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (14s) == sanityn test 55c: rename vs unlink orphan target dir == 06:25:38 (1787739938) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (26s) == sanityn test 55d: rename file vs link ================= 06:26:04 (1787739964) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 06:26:20 (1787739980) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 06:26:25 (1787739985) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 06:26:44 (1787740004) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 06:27:08 (1787740028) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1787740030 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.53543 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (122s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 06:29:10 (1787740150) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1787740157 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.992473 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (141s) == sanityn test 60: Verify data_version behaviour ======== 06:31:31 (1787740291) PASS 60 (12s) == sanityn test 70a: cd directory && rm directory ======== 06:31:43 (1787740303) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:31:53 (1787740313) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:32:03 (1787740323) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0168418 s, 3.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:32:08 (1787740328) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0183638 s, 3.6 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (3s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:32:12 (1787740332) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:32:16 (1787740336) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0351563 s, 1.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:32:21 (1787740341) llite.lustre-ffff977c47b58800.xattr_cache=1 llite.lustre-ffff977c517f0000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (9s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:32:30 (1787740350) llite.lustre-ffff977c47b58800.xattr_cache=1 llite.lustre-ffff977c517f0000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (10s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:32:40 (1787740360) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0403694 s, 25.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0233075 s, 43.9 kB/s 231697: get lock2 231698: get lock1 231697: done 231697: get lock1 231698: done 231698: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231698: cannot get lock2: Resource deadlock avoided 231698: exit rc=0 231697: done 231697: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:32:53 (1787740373) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128463 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116918 s, 9.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232281 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (16s) == sanityn test 76: Verify MDT open_files listing ======== 06:33:10 (1787740390) opening files: ........ closing files: ........ PASS 76 (92s) == sanityn test 77a: check FIFO NRS policy =============== 06:34:42 (1787740482) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465148 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662562 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104921 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12511 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115714 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300908 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262328 s, 4.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295345 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314236 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320567 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23967 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273978 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295482 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386886 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331302 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253592 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41792 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456044 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495195 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163536 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105546 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712048 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759103 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786142 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803578 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920816 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145072 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10788 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911682 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307926 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114364 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119108 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13458 s, 7.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (12s) == sanityn test 77b: check CRR-N NRS policy ============== 06:34:54 (1787740494) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.554695 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409914 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817567 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138085 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898194 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170293 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113176 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233045 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251784 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232314 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185343 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159426 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117595 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175573 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255299 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546487 s, 19.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157712 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967118 s, 10.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00572491 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00896372 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015716 s, 66.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126404 s, 83.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00960409 s, 109 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181307 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00705418 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0118731 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315301 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155211 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301848 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175214 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153199 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115622 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131232 s, 79.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.618438 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607871 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102399 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120923 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820846 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183395 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126105 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105225 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141884 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208169 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974476 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398624 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136444 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191085 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256817 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134073 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137755 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809597 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777252 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836087 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091413 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124062 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878153 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127644 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977066 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106081 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111767 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112813 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120471 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132111 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177361 s, 5.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156144 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156392 s, 6.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (16s) == sanityn test 77c: check ORR NRS policy ================ 06:35:10 (1787740510) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.768752 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875566 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123751 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223948 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121008 s, 8.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260488 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213103 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251549 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282721 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193614 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376389 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34834 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33495 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327645 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461979 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464373 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.572359 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160343 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00389899 s, 269 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00587807 s, 178 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105772 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109676 s, 95.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219172 s, 47.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120903 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170646 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134006 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226077 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114819 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208387 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141479 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153539 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156992 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082878 s, 12.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.600637 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816444 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940257 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108982 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297313 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318053 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316247 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314878 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311315 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411401 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394884 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369126 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296632 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324949 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.450233 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380278 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.509161 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136456 s, 76.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076466 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00496658 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289705 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554933 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352294 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00459516 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00736453 s, 142 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519381 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351084 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396817 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023326 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022195 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384018 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206058 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230243 s, 45.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.441645 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072037 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131738 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117409 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187713 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30324 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30857 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226422 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212651 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363648 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246158 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278864 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370012 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317077 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3026 s, 3.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333814 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49334 s, 2.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129055 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193054 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788937 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133054 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172853 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166627 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926921 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092242 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114914 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872236 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882301 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133131 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826781 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744873 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703591 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562658 s, 18.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 06:35:30 (1787740530) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0127 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618126 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847805 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130183 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158102 s, 6.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220741 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277522 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310244 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301966 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372313 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393923 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418073 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477654 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486003 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462172 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546602 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.612331 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189058 s, 5.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399352 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185112 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103364 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231588 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337672 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328128 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334345 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183098 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172425 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383562 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297691 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224707 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294354 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601353 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685241 s, 15.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.801335 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126499 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287281 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159672 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312078 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455434 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193161 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157494 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373579 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297435 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402056 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153128 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261104 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367505 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544952 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388273 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523494 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958924 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910468 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834316 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857311 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120529 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116746 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107065 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131719 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11338 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911411 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101261 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124101 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122684 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12619 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154559 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137428 s, 7.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.789421 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104466 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10147 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149995 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286428 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324004 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341088 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228886 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349864 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321754 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289072 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252322 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260885 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35208 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378231 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29729 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416967 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070062 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560279 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12731 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940237 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993426 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1169 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106774 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110087 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136939 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110354 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117022 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130791 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132137 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116557 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133736 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161993 s, 6.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (25s) == sanityn test 77e: check TBF NID nrs policy ============ 06:35:55 (1787740555) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.4@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.4089 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825093 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130199 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249591 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260048 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307719 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257297 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262903 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271921 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289761 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420564 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.388555 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449315 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426568 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496317 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476805 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439236 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171858 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100767 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453338 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876912 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114027 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533053 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138619 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18848 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111735 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131315 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101537 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133484 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103854 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129458 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154784 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155047 s, 6.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465917 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158783 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748814 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127447 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131069 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236582 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180879 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233949 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210414 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218807 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234809 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309035 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287244 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291653 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307453 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323943 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392763 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084454 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147481 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244929 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194729 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137402 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158752 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104638 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127445 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804337 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140499 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802753 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945943 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664035 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800568 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660501 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06055 s, 17.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.414951 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487173 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677968 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933958 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577609 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196641 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276232 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287872 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272166 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274276 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253622 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316944 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336827 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370811 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345273 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408522 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406022 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443521 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00524826 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564008 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274344 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094866 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826449 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114284 s, 9.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0072006 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718086 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630171 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129523 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115125 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415955 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337998 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271425 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432277 s, 24.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.354036 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709521 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803642 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223001 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19514 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230953 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251799 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287844 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25523 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302549 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298667 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253548 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290414 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=7 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330445 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317331 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324409 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333747 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608313 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00757751 s, 138 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117689 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129982 s, 80.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140146 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736987 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809321 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206131 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638868 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120864 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585304 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603779 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648933 s, 16.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537638 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492702 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398846 s, 26.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:36:23 (1787740583) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.328398 s, 51.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316393 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366992 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11677 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937016 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111663 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139906 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125971 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124345 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149443 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153521 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159047 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123178 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133154 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14267 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149657 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159865 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448345 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758001 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041421 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535591 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438072 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560806 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570835 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092782 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464309 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485384 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507526 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559943 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573259 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617896 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626412 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653913 s, 16.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291156 s, 57.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987382 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866195 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209985 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200446 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102854 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107478 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22247 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254163 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243217 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294799 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276971 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370579 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186713 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276424 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249684 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411043 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752723 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480091 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438335 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548581 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473563 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00996113 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990604 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532324 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477708 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486111 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203793 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609898 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151543 s, 69.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00508161 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649611 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224894 s, 46.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484846 s, 34.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651461 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105347 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108313 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966667 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1048 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166356 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192033 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202532 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256521 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236013 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214583 s, 4.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279675 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275453 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249732 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293919 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346885 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354758 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874967 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964946 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459787 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459527 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510586 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527439 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612246 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192455 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118886 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817755 s, 12.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858861 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802205 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834219 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853936 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865099 s, 12.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.665074 s, 25.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325641 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998579 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071003 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157692 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194467 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162465 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173254 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162155 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214664 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27721 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312535 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288089 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397232 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288472 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307199 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378837 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=7 execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676763 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06519 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102859 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812212 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213653 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859254 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586835 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101795 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10252 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126827 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165564 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980152 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133477 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137516 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190634 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100448 s, 104 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (30s) == sanityn test 77g: Change TBF type directly ============ 06:36:53 (1787740613) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (12s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:37:06 (1787740626) oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 PASS 77h (11s) == sanityn test 77i: Change rank of TBF rule ============= 06:37:17 (1787740637) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:37:40 (1787740660) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.724702 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134473 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140936 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107525 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225368 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277368 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226132 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218869 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295049 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311595 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368534 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323275 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364073 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409968 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311716 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321389 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317862 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225462 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226979 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254065 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206609 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212921 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268036 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24431 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244946 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220614 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253928 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275932 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279732 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259623 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291944 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27987 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28249 s, 3.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.1347 s, 8.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8123 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (71s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:38:52 (1787740732) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29667 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813596 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122745 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133342 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161445 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197674 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479267 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149904 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180674 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190877 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215449 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272109 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274162 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3349 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308269 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233796 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236859 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413726 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424384 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354003 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444536 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450892 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486605 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364417 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419025 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423366 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483232 s, 21.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453876 s, 23.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445719 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578838 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627584 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570365 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504642 s, 20.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1346 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9766 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33093 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113351 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114607 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147347 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181788 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211491 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212716 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184627 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193705 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206392 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202581 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110286 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060059 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140781 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236892 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358875 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306384 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16873 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169505 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851876 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170784 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172485 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203757 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175457 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128487 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783404 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131856 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140187 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284852 s, 36.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100066 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036842 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203367 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207705 s, 5.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0094 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7848 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (156s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 06:41:28 (1787740888) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.34958 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990512 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120569 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134309 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119851 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149962 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151827 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195271 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194542 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183613 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204977 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2123 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226982 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213523 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27697 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295355 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32077 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755315 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554733 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722308 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834746 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815944 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621347 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985121 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089143 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10515 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101657 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11124 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984703 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093901 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107857 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125633 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998318 s, 10.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.922 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9673 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.34387 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956313 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238843 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237252 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106543 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144382 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337219 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17806 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157584 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352762 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101552 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2185 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225994 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253696 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245607 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268794 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1907 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128494 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110884 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138574 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118742 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858743 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950614 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167528 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144193 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145397 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104083 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187275 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116215 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170899 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200923 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197594 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198401 s, 5.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.4535 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9909 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (155s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 06:44:03 (1787741043) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.49158 s, 34.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029881 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193618 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415425 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139266 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132147 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142225 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165287 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166162 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151387 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178707 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181992 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173418 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173818 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19548 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209271 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207301 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265565 s, 39.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297514 s, 35.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00865808 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309626 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232623 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038315 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780286 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343173 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379549 s, 27.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187087 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722919 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418567 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790829 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363064 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130753 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138186 s, 75.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.71731 s, 13.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0309 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.4@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.4@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.708952 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195096 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142542 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175327 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108445 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133099 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252008 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103013 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972037 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277109 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119675 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111926 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221448 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267434 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332896 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333052 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452945 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440387 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430324 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538934 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108096 s, 97.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818892 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420014 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831793 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703521 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710988 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592524 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244092 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763592 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502982 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610218 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198156 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645735 s, 16.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.47247 s, 11.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92631 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.4@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.542816 s, 30.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112501 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143736 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328878 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317834 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346221 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250624 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325197 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292418 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39494 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396785 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456429 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421599 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336098 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463422 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437612 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478443 s, 2.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624428 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119846 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500295 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324468 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516021 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701659 s, 14.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656433 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701188 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907737 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756582 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956767 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103952 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892123 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893703 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922255 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994756 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.47952 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3469 s, 19.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.816067 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549281 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081895 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467935 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857224 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121405 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117781 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123876 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17856 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124612 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143449 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204867 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22481 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240671 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202857 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169378 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272286 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266868 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783125 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792578 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796207 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556071 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310835 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327733 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360227 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449287 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369201 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05241 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663527 s, 158 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544617 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500996 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557995 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065534 s, 16.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5843 s, 9.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96517 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5478 s, 9.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93534 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24566 s, 13.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563044 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380667 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737648 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223993 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27968 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326609 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313367 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346725 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38736 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.480852 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493259 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466569 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468506 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527651 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502069 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548156 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829158 s, 12.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114354 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919623 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615638 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666535 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656856 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113138 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115075 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688706 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728623 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107713 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147189 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216181 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115808 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119642 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124954 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 22.2554 s, 4.7 MB/s Write runtime is 24 s, speed is 4.166666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6172 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.551495 s, 30.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555276 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116355 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481253 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189529 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255857 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160884 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249808 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280371 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161082 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195456 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191211 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331008 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301579 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318509 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284622 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262708 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769215 s, 136 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737784 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776966 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994364 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760339 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960979 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710812 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110973 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117145 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835693 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977746 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113493 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871307 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125689 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986544 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115272 s, 9.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.77408 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89955 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.788133 s, 21.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680528 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116929 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189498 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169895 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177573 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209798 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199845 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150955 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167678 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157548 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177937 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198851 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191787 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207383 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227583 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200769 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060991 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655149 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066153 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563754 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644891 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791197 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070445 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599037 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105803 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106012 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701009 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843499 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888799 s, 11.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963618 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915825 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100134 s, 10.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9887 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0033 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7395 s, 8.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92478 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (439s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 06:51:22 (1787741482) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.64562 s, 26.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573901 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812693 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191485 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223333 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210198 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265103 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277243 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320133 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278078 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265064 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321298 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30811 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324164 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344785 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319609 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330413 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297297 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364065 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381854 s, 27.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093084 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119087 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994963 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10288 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128229 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102414 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105399 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109018 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724996 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106794 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631736 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113164 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669659 s, 15.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.26985 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88364 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (54s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 06:52:16 (1787741536) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.4@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.4@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5895 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107417 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268892 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176825 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182884 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16068 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211289 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211468 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520059 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174631 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198924 s, 5.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251974 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284553 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285836 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363774 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432337 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435271 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101901 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167425 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151189 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153497 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185869 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154782 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186206 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196202 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17177 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215077 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194016 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172643 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1831 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184297 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193302 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199887 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=6 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5431 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97222 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (54s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 06:53:10 (1787741590) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.4@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.700325 s, 24.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12094 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131416 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118117 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143363 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255243 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251939 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22793 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290123 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338654 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268974 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281008 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322317 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350918 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348723 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387774 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405204 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557037 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00584756 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172344 s, 60.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627961 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137347 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315555 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148066 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128466 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138052 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105032 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107199 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719042 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065657 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856609 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176908 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826683 s, 12.7 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.87607 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.08244 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (42s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 06:53:53 (1787741633) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.662345 s, 25.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832779 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108401 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175637 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208336 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258936 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24482 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262703 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220456 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217488 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259451 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251147 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26879 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270755 s, 3.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306974 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319466 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329376 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176978 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145788 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286246 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885523 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125054 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122564 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108202 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860339 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171972 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170595 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424376 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436377 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963326 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181488 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670205 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514295 s, 20.4 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5512 s, 8.4 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98815 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9171 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0573 s, 10.4 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (102s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 06:55:36 (1787741736) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.05625 s, 15.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662919 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080195 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948135 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940867 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115603 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237945 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195372 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201975 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202754 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286429 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30084 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267387 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312116 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301212 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296748 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325559 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741958 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025872 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666072 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300309 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901352 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661542 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320265 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725054 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958205 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997127 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905495 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103091 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103714 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766138 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795408 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877585 s, 11.9 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.799 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9365 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (75s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 06:56:51 (1787741811) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.774765 s, 21.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105969 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115726 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10409 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215871 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281536 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228552 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247694 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282495 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403955 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305914 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407534 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332285 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381967 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359389 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356975 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482794 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff977c517f0000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff977c517f0000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063905 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168275 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199395 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216174 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032667 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100149 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145684 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179675 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837827 s, 125 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127894 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116251 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00615622 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241324 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144157 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129984 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180692 s, 5.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.2646 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0808 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg304-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.86617 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111846 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986834 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136182 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101678 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270784 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201999 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168284 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1513 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440396 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324487 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252963 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308918 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402837 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255364 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484528 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188766 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195653 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768936 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119937 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179069 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134334 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143559 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213754 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215222 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194155 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172313 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14801 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186398 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153199 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201924 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210559 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197624 s, 5.3 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0705 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.923 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.5215 s, 8.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96963 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (159s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 06:59:30 (1787741970) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg304-client.virtnet /mnt/lustre (opts:) Stopping client oleg304-client.virtnet /mnt/lustre opts: Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg304-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg304-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg304-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.4,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg304-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.4,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg304-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.4,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/68741/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.4,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg304-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg304-server: oleg304-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre Starting client oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre Started clients oleg304-client.virtnet: 192.168.203.104@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Starting client oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 Started clients oleg304-client.virtnet: 192.168.203.104@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff977c42d75800.idle_timeout=debug osc.lustre-OST0000-osc-ffff977c7adf9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c42d75800.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c7adf9800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.422537 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208712 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692298 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744931 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043273 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184504 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18686 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202595 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190841 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212937 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219997 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284975 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279241 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229252 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263088 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293617 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342862 s, 3.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731738 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437931 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300035 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515299 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427528 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472214 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885801 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516922 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642867 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163051 s, 64.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673514 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626714 s, 16.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133344 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622891 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886388 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00798916 s, 131 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.37374 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52566 s, 19.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.711867 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617056 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11041 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117927 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122059 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263658 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29187 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290429 s, 3.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275385 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378791 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352614 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329237 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305614 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35502 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365963 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458352 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475499 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126827 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161585 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172392 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165794 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135985 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118301 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147252 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213338 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173132 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232082 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232916 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240211 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198072 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200253 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174265 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218958 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=7 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0779 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91169 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (212s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 07:03:02 (1787742182) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (20s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 07:03:22 (1787742202) On MGS 192.168.203.104, active = nodemap.active=1 On MGS 192.168.203.104, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.634345 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112676 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110353 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127889 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217307 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238024 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310493 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232923 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255259 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226203 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411944 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373343 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436513 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375734 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427941 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371553 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42712 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483756 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416015 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.50423 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49059 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465685 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43144 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483471 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.544913 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493813 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470441 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499873 s, 2.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4893 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529768 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.527942 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532495 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.547513 s, 1.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.5974 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.69082 s, 18.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.104, TBF.id = On MGS 192.168.203.104, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.609837 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644717 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572727 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894409 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746098 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956597 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143769 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156087 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672054 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164052 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146544 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185765 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224614 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220419 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185878 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209466 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219588 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694403 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166112 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494475 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265838 s, 39.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295961 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519698 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532352 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550803 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414446 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536806 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162759 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683604 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058766 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592524 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693703 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098275 s, 107 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg304-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.20524 s, 12.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.95891 s, 26.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg304-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 On MGS 192.168.203.104, active = nodemap.active=1 On MGS 192.168.203.104, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.714544 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667761 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522537 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542926 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122741 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121689 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282502 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320594 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253645 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338334 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319956 s, 3.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351151 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312573 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34283 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37819 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316822 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417318 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17492 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151068 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21115 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159102 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146437 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186556 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211572 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207685 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207672 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202666 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18049 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230492 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222944 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22853 s, 4.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192424 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194439 s, 5.4 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.91028 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95354 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.104, TBF.id = On MGS 192.168.203.104, active = nodemap.active=0 PASS 77kj (184s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 07:06:27 (1787742387) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (10s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 07:06:37 (1787742397) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg304-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.52541 s, 297 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg304-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.066 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 07:07:34 (1787742454) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.539317 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954243 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129929 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148479 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190112 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103072 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107237 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146511 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133023 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108648 s, 9.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff977c7adf9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310541 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276105 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358866 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292702 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375268 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377631 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362308 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff977c7adf9800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799034 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747406 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058605 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083167 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874611 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093986 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079108 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750704 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073484 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116809 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962792 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094744 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985926 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921298 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107549 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120899 s, 8.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80443 s, 11.9 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32913 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.747325 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571015 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077927 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672067 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737122 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221278 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244185 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337928 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296034 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279432 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364162 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324197 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368384 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35939 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368101 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38465 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433566 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425839 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382749 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391115 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38987 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398129 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465015 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481982 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413183 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454322 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423544 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466851 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501626 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485143 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435689 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462087 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.49095 s, 2.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg304-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg304-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.55617 s, 11.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25345 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (95s) == sanityn test 77o: Changing rank should not panic ====== 07:09:09 (1787742549) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 07:09:21 (1787742561) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (125s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 07:11:26 (1787742686) mds.MDS.mdt.nrs_policies=tbf oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg304-client: oleg304-server: ssh exited with exit code 36 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg304-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg304-client: oleg304-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (36s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 07:12:02 (1787742722) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273395 kill md_thread xargs PASS 77r (56s) == sanityn test 77s: Check TBF LRU shrinker ============== 07:12:58 (1787742778) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (73s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 07:14:11 (1787742851) oleg304-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg304-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg304-client: oleg304-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (13s) == sanityn test 79: xattr: intent error ================== 07:14:24 (1787742864) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935687 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:14:27.000000000 -0400 Modify: 2026-08-26 07:14:27.000000000 -0400 Change: 2026-08-26 07:14:27.000000000 -0400 Birth: 2026-08-26 07:14:27.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 07:14:44 (1787742884) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 07:14:48 (1787742888) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 07:14:52 (1787742892) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 07:14:56 (1787742896) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (5s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 07:15:01 (1787742901) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (5s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 07:15:06 (1787742906) - open/close 434 (time 1787742926.41 total 10.00 last 43.38) - open/close 968 (time 1787742936.42 total 20.01 last 53.37) - open/close 1542 (time 1787742946.42 total 30.01 last 57.38) - open/close 2216 (time 1787742956.43 total 40.02 last 67.35) total: 2500 open/close in 44.19 seconds: 56.57 ops/second snapshot_time 1787743053.861836289 secs.nsecs start_time 1787742912.088138879 secs.nsecs elapsed_time 141.773697410 secs.nsecs open 2505 samples [usecs] 150 23195 4268469 15962489837 close 2505 samples [usecs] 171 10807 917737 863367487 mknod 2501 samples [usecs] 537 23023 3744520 13039412952 unlink 2500 samples [usecs] 724 68730 6333710 64894982398 mkdir 2 samples [usecs] 3453 5649 9102 43834410 rmdir 2 samples [usecs] 936 6881 7817 48224257 rename 2500 samples [usecs] 518 65220 18634379 290046351631 getattr 3763 samples [usecs] 7 12966 252006 629973780 parallel_rename_file 2500 samples [usecs] 518 65220 18634379 290046351631 crossdir_rename 2500 samples [usecs] 518 65220 18634379 290046351631 readpage 29 samples [usecs] 6923 83474 697606 24596236990 PASS 81d (152s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 07:17:39 (1787743059) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7645 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7645 PASS 82 (6s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 07:17:45 (1787743065) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 07:17:48 (1787743068) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 07:18:01 (1787743081) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1787743082 Finishing test test1 at 1787743087 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 07:18:13 (1787743093) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 07:18:16 (1787743096) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 07:18:20 (1787743100) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 07:18:23 (1787743103) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5818 0x16ba 0x240000400 1 5777 0x1691 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5778 0x1692 0x280000400 0 5819 0x16bb 0x240000400 rc1=0 and rc2=0 PASS 93 (16s) == sanityn test 94: signal vs CP callback race =========== 07:18:39 (1787743119) osc.lustre-OST0000-osc-ffff977c42d75800.idle_timeout=0 osc.lustre-OST0000-osc-ffff977c7adf9800.idle_timeout=0 osc.lustre-OST0001-osc-ffff977c42d75800.idle_timeout=0 osc.lustre-OST0001-osc-ffff977c7adf9800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.305872 s, 26.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98322 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290550 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.100854 s, 40.6 kB/s osc.lustre-OST0000-osc-ffff977c42d75800.idle_timeout=debug osc.lustre-OST0000-osc-ffff977c7adf9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c42d75800.idle_timeout=debug osc.lustre-OST0001-osc-ffff977c7adf9800.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 07:18:57 (1787743137) llite.lustre-ffff977c42d75800.fast_read=0 llite.lustre-ffff977c7adf9800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0703444 s, 29.8 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff977c42d75800.fast_read=1 llite.lustre-ffff977c7adf9800.fast_read=1 PASS 95a (22s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 07:19:19 (1787743159) llite.lustre-ffff977c42d75800.fast_read=0 llite.lustre-ffff977c7adf9800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00726242 s, 1.7 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff977c42d75800.fast_read=1 llite.lustre-ffff977c7adf9800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 07:19:33 (1787743173) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 07:19:37 (1787743177) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0180004 s, 228 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 07:19:46 (1787743186) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207121 s, 10.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 07:19:54 (1787743194) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174935 s, 12.0 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 07:20:02 (1787743202) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00688896 s, 1.7 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 07:20:10 (1787743210) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0100548 s, 407 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 07:20:18 (1787743218) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.010685 s, 383 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 07:20:26 (1787743226) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0426878 s, 96.0 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 07:20:34 (1787743234) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 68 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf68:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (9s) == sanityn test 103: Test size correctness with lockahead ========================================================== 07:20:43 (1787743243) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1787743248 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1787743251 Finishing test test23 at 1787743251 PASS 103 (17s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 07:21:01 (1787743261) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 07:21:06 (1787743266) fail_loc=0x416 PASS 105 (29s) == sanityn test 106a: Verify the btime via statx() ======= 07:21:35 (1787743295) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 07:21:39 (1787743299) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0304014 s, 33.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:21:41.000000000 -0400 Modify: 2026-08-26 07:21:41.000000000 -0400 Change: 2026-08-26 07:21:41.000000000 -0400 Birth: 2026-08-26 07:21:41.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938223 -rw-r--r-- 1 0 0 1787743301 1787743301 1787743301 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:21:41.000000000 -0400 Modify: 2026-08-26 07:21:41.000000000 -0400 Change: 2026-08-26 07:21:41.000000000 -0400 Birth: 2026-08-26 07:21:41.000000000 -0400 1787743301 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 07:21:47 (1787743307) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 07:21:55 (1787743315) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.327224 s, 32.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5785 0x1699 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 07:22:06 (1787743326) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.598177 s, 17.5 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5828 0x16c4 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7645 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (18s) == sanityn test 108a: lseek: parallel updates ============ 07:22:24 (1787743344) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0047773 s, 2.1 MB/s llite.lustre-ffff977c42d75800.hybrid_io=0 llite.lustre-ffff977c7adf9800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.86281 s, 1.7 MB/s PASS 108a (17s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 07:22:42 (1787743362) 192.168.203.104@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre (opts:) 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 192.168.203.104@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.104@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg304-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg304-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: detected 4 online CPUs by sysfs oleg304-server: MODOPTS_LIBCFS= oleg304-server: Force libcfs to create 2 CPU partitions Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre Starting client: oleg304-client.virtnet: -o user_xattr,flock 192.168.203.104@tcp:/lustre /mnt/lustre2 PASS 109 (56s) == sanityn test 110: do not grant another lock on resend ========================================================== 07:23:37 (1787743417) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:23:39.000000000 -0400 Modify: 2026-08-26 07:23:39.000000000 -0400 Change: 2026-08-26 07:23:39.000000000 -0400 Birth: 2026-08-26 07:23:39.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg304-client.virtnet oleg304-client.virtnet PASS 110 (72s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 07:24:49 (1787743489) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 07:24:53 (1787743493) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 07:24:57 (1787743497) lustre-OST0000-osc-ffff977c7adfd800 active. lustre-OST0001-osc-ffff977c7adfd800 active. lustre-OST0000-osc-ffff977c7adfd800 active. lustre-OST0001-osc-ffff977c7adfd800 active. MGC192.168.203.104@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 07:25:06 (1787743506) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 07:25:10 (1787743510) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 07:25:14 (1787743514) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 07:25:18 (1787743518) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0151523 s, 1.4 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.133904 s, 153 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (11s) == sanityn test 200: service remains healthy while able to process request ========================================================== 07:25:29 (1787743529) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1804 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1777 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.267 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2347 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3016 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3463 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1655 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1777 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2043 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1932 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2991 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3388 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3305 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3956 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3714 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.349 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg304-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff977c507f2000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff977c507f2000.ost_server_uuid in IDLE state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9812 sec ============== 07:26:52 (1787743612) === sanityn: start cleanup 07:26:54 (1787743614) === Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg304-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 07:31:09 (1787743869) ===