-----============= acceptance-small: sanityn ============----- Fri Sep 11 06:55:17 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg325-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 06:55:35 (1789124135) === Starting client oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Mount client oleg325-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Started clients oleg325-client.virtnet: 192.168.203.125@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg325-client.virtnet: executing check_config_client /mnt/lustre oleg325-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg325-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9130c65e9800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9130d8d03800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130c65e9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130d8d03800.idle_timeout=debug disable quota as required oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanityn: finish setup 06:56:14 (1789124174) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259634 s, 40.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7627] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 06:56:17 (1789124177) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 06:56:24 (1789124184) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 06:56:31 (1789124191) -rw-r--r-- 1 root root 0 Sep 11 06:56 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 06:56:37 (1789124197) -rw-r--r-- 1 root root 0 Sep 11 06:56 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 06:56:43 (1789124203) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 06:56:50 (1789124210) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 06:56:55 (1789124215) total 1 -rwxrwxrwx 1 root root 0 Sep 11 06:56 f2a -rwxrwxrwx 1 root root 0 Sep 11 06:56 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 11 06:56 f2a -rwxrwxrwx 1 root root 0 Sep 11 06:56 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0111881 s, 45.8 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 06:57:01 (1789124221) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 06:57:05 (1789124225) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.492188 s, 4.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 06:57:11 (1789124231) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 06:57:17 (1789124237) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 06:57:24 (1789124244) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 06:57:29 (1789124249) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 06:57:35 (1789124255) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 06:57:40 (1789124260) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 06:57:47 (1789124267) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 06:57:54 (1789124274) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447143 s, 23.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0811047 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 06:58:00 (1789124280) 1+0 records in 1+0 records out 1 byte copied, 0.00808171 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111784 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0221101 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00762525 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0121704 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00827162 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0098733 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00860256 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00952111 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0105598 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0193605 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0149846 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 06:58:07 (1789124287) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0163073 s, 188 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.03567 s, 115 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000249917 s, 12.3 MB/s PASS 10b (7s) == sanityn test 11: execution of file opened for write should return error ============================================================== 06:58:14 (1789124294) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 305: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 06:58:20 (1789124300) warning: '-runtime' deprecated, use '-t runtime' instead - link 313 (time 1789124313.20 total 10.01 last 31.26) - link 630 (time 1789124323.21 total 20.02 last 31.68) - link 931 (time 1789124333.26 total 30.07 last 29.94) - link 1232 (time 1789124343.26 total 40.07 last 30.10) - link 1534 (time 1789124353.28 total 50.09 last 30.13) - link 1855 (time 1789124363.31 total 60.12 last 32.01) - link 2174 (time 1789124373.32 total 70.13 last 31.89) - link 2488 (time 1789124383.35 total 80.16 last 31.29) - link 2742 (time 1789124393.39 total 90.20 last 25.31) total: 2964 link in 100.01 seconds: 29.64 ops/second using seed 1248267842 running for 100 seconds - stat 5740 (time 1789124315 ; total 11 ; last 11) - stat 10000 (time 1789124323 ; total 19 ; last 8) - stat 15454 (time 1789124334 ; total 30 ; last 11) - stat 20000 (time 1789124343 ; total 39 ; last 9) - stat 25115 (time 1789124354 ; total 50 ; last 11) - stat 27990 (time 1789124365 ; total 61 ; last 11) - stat 30000 (time 1789124373 ; total 69 ; last 8) - stat 32467 (time 1789124384 ; total 80 ; last 11) - stat 34401 (time 1789124395 ; total 91 ; last 11) total: 36001 stats in 100 seconds: 360.010010 stats/second - unlinked 0 (time 1789124354 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2964) error: No such file or directory total: 2963 unlinks in 87 seconds: 34.057472 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21633) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21635) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 07:00:47 (1789124447) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 07:00:55 (1789124455) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 340: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 22818 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 07:01:03 (1789124463) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 23411 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 07:01:11 (1789124471) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 355: 23990 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 07:01:18 (1789124478) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 367: 24572 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 07:01:25 (1789124485) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 378: 25156 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 07:01:33 (1789124493) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (47s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 07:02:20 (1789124540) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.481715 s, 17.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.329 s, 25.5 MB/s Chance of close/open is 1 in 50 Seed set to 4544 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[1] 1789124545.364912 mapwrite 0x73687c thru 0x73bebc (0x5641 bytes) 000005[1] 1789124545.550772 mapwrite 0x8d7e2f thru 0x8e29da (0xabac bytes) truncating to largest ever: 0x617c1e 000006[0] 1789124545.712145 trunc from 0x8e29db to 0x617c1d (0x2cadbd bytes) 000008[1] 1789124545.811890 write 0x0216bd thru 0x0294b2 (0x7df6 bytes) 000009[1] 1789124545.876282 mapread 0x14d896 thru 0x158ed9 (0xb644 bytes) 000011[1] 1789124546.008942 read 0x3fd4e1 thru 0x408251 (0xad71 bytes) 000014[1] 1789124546.056782 read 0x43f346 thru 0x44c78a (0xd445 bytes) 000019[1] 1789124546.188190 mapwrite 0x31b553 thru 0x321e1f (0x68cd bytes) 000020[1] 1789124546.319829 write 0x49763d thru 0x49daa3 (0x6467 bytes) 000021[0] 1789124546.372344 mapwrite 0x28de2a thru 0x293a8f (0x5c66 bytes) 000022[0] 1789124546.586776 trunc from 0x617c1e to 0x59d40d (0x7a810 bytes) truncating to largest ever: 0x94fc2e 000025[0] 1789124546.766566 trunc from 0x59d40e to 0x94fc2d (0x3b2820 bytes) 000026[1] 1789124546.786653 mapwrite 0x9ff058 thru 0x9fffff (0xfa8 bytes) 000027[1] 1789124546.919668 read 0x6ca953 thru 0x6d16af (0x6d5d bytes) 000028[1] 1789124546.935399 read 0x252dad thru 0x25f14d (0xc3a1 bytes) 000029[1] 1789124546.969042 read 0x0691cd thru 0x070b5f (0x7993 bytes) 000030[0] 1789124546.981246 read 0x3d726a thru 0x3e15da (0xa371 bytes) 000031[0] 1789124547.017253 mapwrite 0x3b9eec thru 0x3c5d3d (0xbe52 bytes) 000039[1] 1789124547.427979 mapwrite 0x8153d2 thru 0x81959c (0x41cb bytes) 000040[1] 1789124547.546714 write 0x3529ab thru 0x3556cc (0x2d22 bytes) 000041[1] 1789124547.583193 read 0x790e0f thru 0x794ee1 (0x40d3 bytes) 000042[0] 1789124547.612402 write 0x018b9e thru 0x01be1a (0x327d bytes) 000047[0] 1789124547.727971 mapread 0x0d7f5c thru 0x0db4b8 (0x355d bytes) 000048[0] 1789124547.769897 mapread 0x0bb311 thru 0x0bde63 (0x2b53 bytes) 000049[0] 1789124547.805951 mapread 0x222c96 thru 0x22646d (0x37d8 bytes) 000051[0] 1789124547.865381 read 0x72566e thru 0x72cd49 (0x76dc bytes) 000052[0] 1789124547.885811 trunc from 0xa00000 to 0x3b7657 (0x6489a8 bytes) 000053[0] 1789124547.943148 mapread 0x2a79aa thru 0x2b2105 (0xa75c bytes) 000054[0] 1789124547.976737 write 0x1a8e7b thru 0x1b1ceb (0x8e71 bytes) 000056[0] 1789124548.017376 read 0x30f66e thru 0x31609c (0x6a2f bytes) 000057[0] 1789124548.136042 write 0x1ed561 thru 0x1f7c28 (0xa6c8 bytes) 000059[0] 1789124548.168363 write 0x183628 thru 0x187eeb (0x48c4 bytes) 000060[0] 1789124548.173757 trunc from 0x3b7658 to 0x888fb8 (0x4d1961 bytes) 000062[0] 1789124548.225047 mapwrite 0x0d1d1e thru 0x0dc286 (0xa569 bytes) 000063[0] 1789124548.322075 mapread 0x52a76c thru 0x539ec8 (0xf75d bytes) 000064[1] 1789124548.398415 write 0x965322 thru 0x9736fd (0xe3dc bytes) 000066[1] 1789124548.476678 mapwrite 0x51514a thru 0x524620 (0xf4d7 bytes) 000067[1] 1789124548.718281 mapwrite 0x1004c4 thru 0x106ab9 (0x65f6 bytes) 000068[0] 1789124548.825939 trunc from 0x9736fe to 0x580fa5 (0x3f2758 bytes) 000070[0] 1789124548.907140 read 0x069686 thru 0x06a054 (0x9cf bytes) 000072[0] 1789124548.933739 mapread 0x0d5902 thru 0x0e13c9 (0xbac8 bytes) 000077[0] 1789124549.093747 mapread 0x0ca35c thru 0x0d7ae4 (0xd789 bytes) 000078[1] 1789124549.158620 read 0x2d1c37 thru 0x2e1410 (0xf7da bytes) 000079[0] 1789124549.228593 write 0x9a5a74 thru 0x9ad285 (0x7812 bytes) 000080[1] 1789124549.246744 read 0x736887 thru 0x7421e9 (0xb963 bytes) 000081[1] 1789124549.350119 write 0x2f6ed5 thru 0x300da7 (0x9ed3 bytes) 000082[1] 1789124549.418872 write 0x1b913f thru 0x1c8c94 (0xfb56 bytes) 000083[1] 1789124549.431350 write 0x0634f2 thru 0x063fbf (0xace bytes) 000084[1] 1789124549.436963 mapread 0x3abb20 thru 0x3b1623 (0x5b04 bytes) 000085[1] 1789124549.472544 mapread 0x98ad92 thru 0x9997d8 (0xea47 bytes) 000087[1] 1789124549.519751 trunc from 0x9ad286 to 0x84a21a (0x16306b bytes) 000088[1] 1789124549.618846 trunc from 0x84a21b to 0x489cb3 (0x3c0567 bytes) 000089[0] 1789124549.644302 write 0x7da240 thru 0x7e633b (0xc0fc bytes) 000092[0] 1789124549.748528 write 0x2d4a82 thru 0x2dbef8 (0x7477 bytes) 000095[0] 1789124549.869649 read 0x5b6531 thru 0x5c346a (0xcf3a bytes) 000101[1] 1789124549.956791 write 0x9783d0 thru 0x97e4bf (0x60f0 bytes) 000103[0] 1789124550.024030 trunc from 0x97e4c0 to 0x26ca81 (0x711a3e bytes) 000104[1] 1789124550.106508 mapwrite 0x413602 thru 0x419f76 (0x6975 bytes) 000107[0] 1789124550.438537 mapread 0x266ae9 thru 0x26847a (0x1992 bytes) 000108[1] 1789124550.536155 mapwrite 0x2115b4 thru 0x21ec8f (0xd6dc bytes) 000109[1] 1789124550.747505 trunc from 0x419f77 to 0x4b2f6c (0x98ff6 bytes) 000110[1] 1789124550.773821 trunc from 0x4b2f6d to 0x408acf (0xaa49d bytes) 000119[1] 1789124550.875197 write 0x8369f0 thru 0x83a577 (0x3b88 bytes) 000122[0] 1789124550.938188 mapread 0x236219 thru 0x243384 (0xd16c bytes) 000124[1] 1789124551.028196 write 0x7896ec thru 0x798fee (0xf903 bytes) 000128[0] 1789124551.127682 read 0x493686 thru 0x499c80 (0x65fb bytes) 000129[1] 1789124551.176200 write 0x53e3f2 thru 0x540eb8 (0x2ac7 bytes) 000132[1] 1789124551.230021 mapwrite 0x1298cc thru 0x12dd47 (0x447c bytes) 000133[0] 1789124551.360742 mapwrite 0x53254a thru 0x535cae (0x3765 bytes) 000134[0] 1789124551.585623 trunc from 0x83a578 to 0x8e3c20 (0xa96a9 bytes) 000135[0] 1789124551.678916 read 0x4808be thru 0x48e233 (0xd976 bytes) 000137[1] 1789124551.773178 mapread 0x292624 thru 0x2a1a94 (0xf471 bytes) 000138[1] 1789124551.868268 write 0x6f3e27 thru 0x6fe9f5 (0xabcf bytes) 000139[1] 1789124551.895039 read 0x6f84e8 thru 0x6fad48 (0x2861 bytes) 000140[1] 1789124551.895856 mapwrite 0x58ad6d thru 0x590d0f (0x5fa3 bytes) 000141[1] 1789124551.975168 read 0x1f9346 thru 0x207e00 (0xeabb bytes) 000146[0] 1789124552.075738 mapwrite 0x9d9bee thru 0x9db23e (0x1651 bytes) 000147[0] 1789124552.288846 write 0x4a9349 thru 0x4aaccd (0x1985 bytes) 000148[0] 1789124552.310903 write 0x113fe7 thru 0x123843 (0xf85d bytes) 000149[1] 1789124552.342170 read 0x31854d thru 0x3280c1 (0xfb75 bytes) 000154[1] 1789124552.494034 mapread 0x7bb577 thru 0x7bf896 (0x4320 bytes) 000159[0] 1789124552.613990 mapwrite 0x41edcf thru 0x42a322 (0xb554 bytes) 000162[1] 1789124552.821299 mapread 0x8c5609 thru 0x8cd988 (0x8380 bytes) 000163[0] 1789124552.852713 read 0x4ec08b thru 0x4f98f1 (0xd867 bytes) 000164[0] 1789124552.865332 trunc from 0x9db23f to 0x1bb5bc (0x81fc82 bytes) 000165[0] 1789124552.967318 trunc from 0x1bb5bd to 0x2e66a3 (0x12b0e7 bytes) 000167[1] 1789124553.055100 trunc from 0x2e66a4 to 0x0ee3d8 (0x1f82cb bytes) 000170[0] 1789124553.174998 read 0x0d1480 thru 0x0df307 (0xde88 bytes) 000173[0] 1789124553.283601 trunc from 0x0ee3d9 to 0x945a71 (0x857699 bytes) 000174[1] 1789124553.385930 mapread 0x0e1eb1 thru 0x0eab1f (0x8c6f bytes) 000177[0] 1789124553.563631 mapwrite 0x0cda2f thru 0x0cf5c4 (0x1b96 bytes) 000179[0] 1789124553.760964 mapwrite 0x9feed5 thru 0x9fffff (0x112b bytes) 000182[1] 1789124554.088134 mapwrite 0x2e8311 thru 0x2ef93d (0x762d bytes) 000183[1] 1789124554.263864 read 0x2bab44 thru 0x2bfc78 (0x5135 bytes) 000184[1] 1789124554.303389 mapwrite 0x5fc7b0 thru 0x603f3c (0x778d bytes) 000187[0] 1789124554.430596 mapread 0x5bf06b thru 0x5c32b7 (0x424d bytes) 000189[1] 1789124554.527860 trunc from 0xa00000 to 0x40257d (0x5fda82 bytes) 000190[1] 1789124554.578620 mapread 0x0b6259 thru 0x0b7c89 (0x1a31 bytes) 000193[0] 1789124554.615558 read 0x2aede7 thru 0x2b0390 (0x15aa bytes) 000194[0] 1789124554.631179 write 0x7dd7b0 thru 0x7e8b42 (0xb393 bytes) 000195[1] 1789124554.645290 mapread 0x0d82fb thru 0x0e4445 (0xc14b bytes) 000196[0] 1789124554.678725 write 0x387bd1 thru 0x394fb8 (0xd3e8 bytes) 000197[0] 1789124554.717570 mapwrite 0x607b30 thru 0x6169ea (0xeebb bytes) 000198[0] 1789124554.845652 mapread 0x092f8c thru 0x096a7d (0x3af2 bytes) 000202[1] 1789124554.939000 mapread 0x7bf03d thru 0x7cd516 (0xe4da bytes) 000203[0] 1789124555.010307 read 0x16b54a thru 0x17a301 (0xedb8 bytes) 000206[0] 1789124555.079457 mapwrite 0x4b7534 thru 0x4b8bc5 (0x1692 bytes) 000207[0] 1789124555.162579 read 0x078914 thru 0x07bcd6 (0x33c3 bytes) 000209[0] 1789124555.222427 read 0x057246 thru 0x05ec66 (0x7a21 bytes) 000212[1] 1789124555.234747 write 0x4e27f4 thru 0x4ef7ff (0xd00c bytes) 000214[1] 1789124555.284005 write 0x5c9308 thru 0x5c93a0 (0x99 bytes) 000217[1] 1789124555.349080 trunc from 0x7e8b43 to 0x7dac11 (0xdf31 bytes) 000218[0] 1789124555.415335 mapread 0x7cc517 thru 0x7d01df (0x3cc9 bytes) 000224[1] 1789124555.650719 trunc from 0x7dac12 to 0x1ab024 (0x62fbed bytes) 000225[1] 1789124555.695993 mapread 0x0677f5 thru 0x0689e3 (0x11ef bytes) 000226[0] 1789124555.707624 write 0x5e4680 thru 0x5f2cbf (0xe640 bytes) 000229[1] 1789124555.769765 read 0x43fcd8 thru 0x441c6c (0x1f95 bytes) 000230[1] 1789124555.805901 write 0x4f2d3c thru 0x4f5fa3 (0x3268 bytes) 000231[0] 1789124555.873132 read 0x2a5fa9 thru 0x2a79f7 (0x1a4f bytes) 000233[0] 1789124555.924941 mapread 0x29b729 thru 0x2aa1bd (0xea95 bytes) 000236[0] 1789124556.153379 trunc from 0x5f2cc0 to 0x24afe5 (0x3a7cda bytes) 000237[1] 1789124556.247679 mapread 0x0d86a8 thru 0x0e589a (0xd1f3 bytes) 000238[0] 1789124556.318314 mapread 0x139ce7 thru 0x14973c (0xfa56 bytes) 000239[1] 1789124556.374619 mapwrite 0x889686 thru 0x89209a (0x8a15 bytes) 000241[1] 1789124556.543457 mapread 0x71d759 thru 0x72b30f (0xdbb7 bytes) 000244[0] 1789124556.686612 mapread 0x140f3c thru 0x1463d9 (0x549e bytes) 000245[1] 1789124556.749352 write 0x0a6bd5 thru 0x0ac194 (0x55c0 bytes) 000247[1] 1789124556.811401 write 0x8acb2b thru 0x8ae0ac (0x1582 bytes) 000248[0] 1789124556.812861 mapwrite 0x699841 thru 0x6a7712 (0xded2 bytes) 000250[0] 1789124556.988585 mapwrite 0x1856f0 thru 0x190969 (0xb27a bytes) 000252[0] 1789124557.262549 mapread 0x38c81b thru 0x39a2d8 (0xdabe bytes) 000253[0] 1789124557.338879 mapwrite 0x1ae93e thru 0x1b4b97 (0x625a bytes) 000255[0] 1789124557.492937 mapread 0x8417b6 thru 0x84702f (0x587a bytes) 000256[1] 1789124557.570608 mapwrite 0x9b1978 thru 0x9b47b0 (0x2e39 bytes) 000257[0] 1789124557.757122 read 0x1be5a8 thru 0x1c3308 (0x4d61 bytes) 000258[1] 1789124557.851853 read 0x4b866a thru 0x4c5dde (0xd775 bytes) 000259[0] 1789124557.866329 write 0x62cbe1 thru 0x62dd9a (0x11ba bytes) 000261[1] 1789124557.892726 mapwrite 0x0281ee thru 0x0291e2 (0xff5 bytes) 000266[0] 1789124558.108110 write 0x5a4285 thru 0x5b0f75 (0xccf1 bytes) 000268[1] 1789124558.158837 mapread 0x7b81f4 thru 0x7bc5fa (0x4407 bytes) 000270[1] 1789124558.254068 mapread 0x398117 thru 0x3a2baf (0xaa99 bytes) 000271[0] 1789124558.291121 mapread 0x29f283 thru 0x29f89c (0x61a bytes) 000272[1] 1789124558.318899 read 0x68ea1d thru 0x693621 (0x4c05 bytes) 000277[0] 1789124558.331291 mapwrite 0x8c19c2 thru 0x8cffc2 (0xe601 bytes) 000278[0] 1789124558.436262 write 0x19e909 thru 0x1a06f7 (0x1def bytes) 000279[1] 1789124558.446431 mapread 0x24d146 thru 0x24e5a3 (0x145e bytes) 000281[1] 1789124558.486619 mapwrite 0x6205c4 thru 0x628ca2 (0x86df bytes) 000282[1] 1789124558.573638 trunc from 0x9b47b1 to 0x18b87b (0x828f35 bytes) 000283[0] 1789124558.643914 write 0x64dcfa thru 0x64fbf1 (0x1ef8 bytes) 000285[0] 1789124558.712597 mapread 0x1d9cef thru 0x1e87b3 (0xeac5 bytes) 000286[0] 1789124558.882299 mapwrite 0x31f482 thru 0x32ad14 (0xb893 bytes) 000291[0] 1789124559.144052 read 0x1c343f thru 0x1c9b1f (0x66e1 bytes) 000292[0] 1789124559.175344 mapread 0x05f8a6 thru 0x06d5aa (0xdd05 bytes) 000293[0] 1789124559.224421 trunc from 0x64fbf2 to 0x072261 (0x5dd990 bytes) 000294[0] 1789124559.247055 write 0x66d65a thru 0x675def (0x8796 bytes) 000296[1] 1789124559.292065 write 0x0e7d26 thru 0x0ecff1 (0x52cc bytes) 000300[1] 1789124559.416771 mapwrite 0x14169f thru 0x14cc1c (0xb57e bytes) 000301[0] 1789124559.778136 mapwrite 0x6f5a78 thru 0x701eb5 (0xc43e bytes) 000303[0] 1789124560.008099 mapread 0x0f8c70 thru 0x0feeec (0x627d bytes) 000308[0] 1789124560.132351 trunc from 0x701eb6 to 0x2e434f (0x41db66 bytes) 000309[1] 1789124560.158270 write 0x748269 thru 0x7562f8 (0xe090 bytes) 000310[0] 1789124560.245633 mapread 0x58459f thru 0x58ca43 (0x84a5 bytes) 000311[0] 1789124560.349291 write 0x6bec88 thru 0x6c92ad (0xa626 bytes) 000316[0] 1789124560.517453 read 0x32417e thru 0x324cd0 (0xb53 bytes) 000317[0] 1789124560.573384 write 0x99a615 thru 0x99b4cc (0xeb8 bytes) 000325[0] 1789124560.726928 read 0x182e2f thru 0x18e57b (0xb74d bytes) 000328[0] 1789124560.792989 mapread 0x715195 thru 0x724557 (0xf3c3 bytes) 000331[0] 1789124560.913821 mapread 0x50d769 thru 0x50fa3d (0x22d5 bytes) 000333[0] 1789124560.985699 mapread 0x084138 thru 0x088d7f (0x4c48 bytes) 000334[0] 1789124561.028679 read 0x32883c thru 0x32c060 (0x3825 bytes) 000335[1] 1789124561.059167 mapread 0x8ba212 thru 0x8bdc86 (0x3a75 bytes) 000337[0] 1789124561.152463 write 0x982d9b thru 0x990b10 (0xdd76 bytes) 000338[0] 1789124561.197301 mapread 0x3fd22c thru 0x40a98c (0xd761 bytes) 000339[0] 1789124561.260470 mapwrite 0x267108 thru 0x274b79 (0xda72 bytes) 000341[1] 1789124561.428906 write 0x4ce805 thru 0x4d90c0 (0xa8bc bytes) 000343[1] 1789124561.525389 trunc from 0x99b4cd to 0x80eb3f (0x18c98d bytes) 000347[0] 1789124561.593666 mapwrite 0x0813f7 thru 0x0844b4 (0x30be bytes) 000349[0] 1789124561.704137 mapread 0x6e86cf thru 0x6f06fd (0x802f bytes) 000353[0] 1789124561.772167 write 0x197805 thru 0x1a3092 (0xb88e bytes) 000355[1] 1789124561.799808 mapwrite 0x01dfdc thru 0x029acf (0xbaf4 bytes) 000357[0] 1789124561.921327 write 0x29c5bb thru 0x29cf86 (0x9cc bytes) 000361[0] 1789124561.961344 read 0x58af65 thru 0x593f7e (0x901a bytes) 000362[1] 1789124561.985028 write 0x634b5e thru 0x63690e (0x1db1 bytes) 000364[1] 1789124562.053965 read 0x6cbce1 thru 0x6d67c5 (0xaae5 bytes) 000365[1] 1789124562.077933 read 0x78fbf7 thru 0x79df82 (0xe38c bytes) 000366[1] 1789124562.101783 read 0x107b03 thru 0x1149cc (0xceca bytes) 000370[0] 1789124562.208663 mapread 0x1d7667 thru 0x1d8856 (0x11f0 bytes) 000371[1] 1789124562.287140 trunc from 0x80eb40 to 0x524b52 (0x2e9fed bytes) 000372[0] 1789124562.365407 read 0x0b71c0 thru 0x0bb221 (0x4062 bytes) 000374[1] 1789124562.425776 mapread 0x37948f thru 0x382de8 (0x995a bytes) 000375[1] 1789124562.486715 trunc from 0x524b53 to 0x19a956 (0x38a1fc bytes) 000376[0] 1789124562.582798 mapwrite 0x6b0eb9 thru 0x6b3ddc (0x2f24 bytes) 000377[1] 1789124562.797488 write 0x141732 thru 0x14dcd5 (0xc5a4 bytes) 000380[1] 1789124562.903412 mapread 0x371f65 thru 0x38119f (0xf23b bytes) 000381[0] 1789124562.964859 mapwrite 0x9cdef3 thru 0x9d76a2 (0x97b0 bytes) 000383[1] 1789124563.129173 mapwrite 0x2cff50 thru 0x2ddfa3 (0xe054 bytes) 000384[0] 1789124563.246017 trunc from 0x9d76a3 to 0x63cc24 (0x39aa7e bytes) 000385[0] 1789124563.279649 mapwrite 0x3dc1f2 thru 0x3eb828 (0xf637 bytes) 000386[0] 1789124563.373870 trunc from 0x63cc25 to 0x15f5e5 (0x4dd63f bytes) 000388[1] 1789124563.396977 mapwrite 0x381606 thru 0x385d12 (0x470d bytes) 000391[1] 1789124563.515936 mapwrite 0x2a427d thru 0x2b0abe (0xc842 bytes) 000395[0] 1789124563.632987 mapread 0x1606f6 thru 0x162569 (0x1e74 bytes) 000401[0] 1789124563.829067 read 0x20ec2e thru 0x212e93 (0x4266 bytes) 000402[0] 1789124563.942598 mapwrite 0x7c6c72 thru 0x7caf95 (0x4324 bytes) 000403[1] 1789124564.293539 mapwrite 0x2e086d thru 0x2e2a9d (0x2231 bytes) 000405[0] 1789124564.490815 mapwrite 0x21503a thru 0x219bb4 (0x4b7b bytes) 000406[0] 1789124564.605145 mapread 0x39e118 thru 0x3a89ec (0xa8d5 bytes) 000410[0] 1789124564.786305 mapwrite 0x68ab7d thru 0x691e63 (0x72e7 bytes) 000412[1] 1789124564.910803 mapread 0x379126 thru 0x3886a5 (0xf580 bytes) 000413[1] 1789124565.005913 trunc from 0x7caf96 to 0x349a50 (0x481545 bytes) 000414[1] 1789124565.132898 mapread 0x0dbd24 thru 0x0e8a15 (0xccf2 bytes) 000415[0] 1789124565.237053 read 0x283d6f thru 0x284306 (0x598 bytes) 000417[0] 1789124565.412301 trunc from 0x349a51 to 0x0b3265 (0x2967eb bytes) 000419[0] 1789124565.540310 mapwrite 0x50c8ee thru 0x50e829 (0x1f3c bytes) 000421[0] 1789124565.681863 mapread 0x04ec00 thru 0x05a1d4 (0xb5d5 bytes) 000423[0] 1789124565.783658 read 0x0339e0 thru 0x03cd83 (0x93a4 bytes) 000424[0] 1789124565.794466 read 0x2fef78 thru 0x30cedd (0xdf66 bytes) 000429[1] 1789124566.005118 mapread 0x29c869 thru 0x2a0548 (0x3ce0 bytes) 000432[1] 1789124566.113690 trunc from 0x50e82a to 0x2ec084 (0x2227a5 bytes) 000433[0] 1789124566.169149 mapread 0x1303c3 thru 0x13bff6 (0xbc34 bytes) truncating to largest ever: 0x9dc946 000434[0] 1789124566.231000 trunc from 0x2ec085 to 0x9dc945 (0x6f08c1 bytes) 000435[0] 1789124566.285626 read 0x5a2f6c thru 0x5a5a31 (0x2ac6 bytes) 000436[1] 1789124566.291901 write 0x29c2b1 thru 0x2a26cd (0x641d bytes) 000437[0] 1789124566.331718 write 0x06b0c5 thru 0x074cce (0x9c0a bytes) 000438[1] 1789124566.398822 read 0x40fb4e thru 0x41a977 (0xae2a bytes) 000439[1] 1789124566.472981 mapwrite 0x92b155 thru 0x931f0b (0x6db7 bytes) 000440[0] 1789124566.770307 mapwrite 0x5d030f thru 0x5d685f (0x6551 bytes) 000441[0] 1789124567.003118 read 0x64a323 thru 0x64ad53 (0xa31 bytes) 000442[1] 1789124567.041005 mapread 0x21855d thru 0x21dab9 (0x555d bytes) 000443[1] 1789124567.113031 mapwrite 0x7cd7b8 thru 0x7d5eeb (0x8734 bytes) 000446[1] 1789124567.277999 write 0x62550a thru 0x62908b (0x3b82 bytes) 000447[1] 1789124567.292208 mapread 0x048ce6 thru 0x04f3a3 (0x66be bytes) 000449[1] 1789124567.321752 mapwrite 0x2af035 thru 0x2b2de6 (0x3db2 bytes) 000450[1] 1789124567.454810 write 0x7b5f9c thru 0x7c21fc (0xc261 bytes) 000451[1] 1789124567.474893 mapwrite 0x315372 thru 0x31fe8c (0xab1b bytes) 000452[1] 1789124567.556942 read 0x0de607 thru 0x0e020f (0x1c09 bytes) 000455[1] 1789124567.680172 read 0x8d4a7b thru 0x8de3c4 (0x994a bytes) 000459[0] 1789124567.867601 trunc from 0x9dc946 to 0x1f341c (0x7e9529 bytes) 000461[0] 1789124567.951846 mapwrite 0x6646ef thru 0x672f4d (0xe85f bytes) 000464[0] 1789124568.139156 mapread 0x006ba6 thru 0x0097c4 (0x2c1f bytes) 000465[0] 1789124568.199973 write 0x937087 thru 0x9380f4 (0x106e bytes) 000467[1] 1789124568.263495 mapwrite 0x4da168 thru 0x4df58b (0x5424 bytes) 000468[1] 1789124568.385889 write 0x08e34e thru 0x08e861 (0x514 bytes) 000469[0] 1789124568.409816 trunc from 0x9380f5 to 0x538d41 (0x3ff3b3 bytes) 000470[1] 1789124568.491116 mapread 0x4c23d5 thru 0x4c4f76 (0x2ba2 bytes) 000471[0] 1789124568.548191 write 0x465bd6 thru 0x472caf (0xd0da bytes) 000472[1] 1789124568.588279 mapread 0x219d6a thru 0x223011 (0x92a8 bytes) 000473[0] 1789124568.652730 trunc from 0x538d42 to 0x38869a (0x1b06a7 bytes) 000474[0] 1789124568.715847 trunc from 0x38869b to 0x088c13 (0x2ffa87 bytes) 000475[0] 1789124568.737187 trunc from 0x088c14 to 0x141594 (0xb8981 bytes) 000476[0] 1789124568.775162 mapwrite 0x14fee3 thru 0x158bd0 (0x8cee bytes) 000479[0] 1789124569.010871 trunc from 0x158bd1 to 0x8b9949 (0x760d79 bytes) 000480[1] 1789124569.041528 mapwrite 0x1e4e21 thru 0x1eac57 (0x5e37 bytes) 000482[1] 1789124569.167349 mapread 0x62d489 thru 0x635163 (0x7cdb bytes) 000483[0] 1789124569.213186 trunc from 0x8b994a to 0x508499 (0x3b14b0 bytes) 000485[0] 1789124569.329816 trunc from 0x50849a to 0x6038d6 (0xfb43d bytes) 000486[0] 1789124569.386775 mapread 0x589f1d thru 0x5960a0 (0xc184 bytes) 000490[1] 1789124569.514296 write 0x16bead thru 0x16df63 (0x20b7 bytes) 000500[1] 1789124569.749770 mapread 0x53962f thru 0x53f00b (0x59dd bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 4569 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1789124570.696936 mapwrite 0x034000 thru 0x038b15 (0x4b16 bytes) truncating to largest ever: 0x9b9a1f 000006[1] 1789124570.950335 trunc from 0x038b16 to 0x9b9a1e (0x980f09 bytes) 000008[0] 1789124570.978323 mapread 0x7d2000 thru 0x7e0f82 (0xef83 bytes) 000009[0] 1789124571.063072 mapread 0x609000 thru 0x616118 (0xd119 bytes) 000012[1] 1789124571.107660 write_OD 0x794000 thru 0x79bfff (0x8000 bytes) 000015[0] 1789124571.165052 trunc from 0x9b9a1f to 0x389677 (0x6303a7 bytes) 000016[0] 1789124571.201407 trunc from 0x389678 to 0x2b2766 (0xd6f11 bytes) 000019[0] 1789124571.217378 write_OD 0x54f000 thru 0x55bfff (0xd000 bytes) 000020[0] 1789124571.269140 mapwrite 0x241000 thru 0x24c92d (0xb92e bytes) 000021[1] 1789124571.369320 read_OD 0x4dd000 thru 0x4e7fff (0xb000 bytes) 000024[1] 1789124571.400888 trunc from 0x55c000 to 0x178cb5 (0x3e334a bytes) 000025[0] 1789124571.478363 mapwrite 0x0d9000 thru 0x0e2ba7 (0x9ba8 bytes) 000027[0] 1789124571.660627 trunc from 0x178cb6 to 0x6a2ab6 (0x529e01 bytes) 000029[0] 1789124571.699399 trunc from 0x6a2ab7 to 0x6d3c4e (0x31198 bytes) 000031[1] 1789124571.718839 trunc from 0x6d3c4f to 0x11b060 (0x5b8bee bytes) 000033[1] 1789124571.808027 read_OD 0x0c5000 thru 0x0c7fff (0x3000 bytes) 000038[0] 1789124571.863260 mapwrite 0x866000 thru 0x86d1ae (0x71af bytes) 000041[1] 1789124572.037338 trunc from 0x86d1af to 0x7609ec (0x10c7c2 bytes) 000042[0] 1789124572.066790 mapread 0x4e9000 thru 0x4ec504 (0x3505 bytes) 000047[1] 1789124572.134507 read_OD 0x3da000 thru 0x3ddfff (0x4000 bytes) 000048[0] 1789124572.151152 mapwrite 0x8d1000 thru 0x8e0110 (0xf111 bytes) 000050[1] 1789124572.289874 read_OD 0x611000 thru 0x619fff (0x9000 bytes) 000056[0] 1789124572.330725 write_OD 0x33e000 thru 0x33efff (0x1000 bytes) 000058[0] 1789124572.386655 mapwrite 0x5a8000 thru 0x5b3734 (0xb735 bytes) 000061[0] 1789124572.512906 mapread 0x290000 thru 0x292d3c (0x2d3d bytes) 000063[0] 1789124572.550998 mapread 0x6a9000 thru 0x6b06b2 (0x76b3 bytes) 000070[0] 1789124572.622496 mapread 0x273000 thru 0x27dfa2 (0xafa3 bytes) 000073[0] 1789124572.670949 mapwrite 0x1b1000 thru 0x1bfd8a (0xed8b bytes) 000076[0] 1789124572.796905 write_OD 0x7af000 thru 0x7b2fff (0x4000 bytes) 000077[1] 1789124572.877153 read_OD 0x8d9000 thru 0x8d9fff (0x1000 bytes) 000079[1] 1789124572.905182 write_OD 0x429000 thru 0x434fff (0xc000 bytes) 000080[1] 1789124572.971348 mapwrite 0x43c000 thru 0x44b0c1 (0xf0c2 bytes) 000085[1] 1789124573.173841 mapwrite 0x399000 thru 0x3a2c3a (0x9c3b bytes) 000090[0] 1789124573.328555 mapwrite 0x2c7000 thru 0x2d2336 (0xb337 bytes) 000091[1] 1789124573.489627 read_OD 0x072000 thru 0x078fff (0x7000 bytes) 000094[1] 1789124573.535349 mapwrite 0x98e000 thru 0x99bd45 (0xdd46 bytes) 000095[0] 1789124573.716315 trunc from 0x99bd46 to 0x2ff83d (0x69c508 bytes) 000097[1] 1789124573.764704 read_OD 0x0cf000 thru 0x0d7fff (0x9000 bytes) 000098[1] 1789124573.849025 mapread 0x152000 thru 0x15683c (0x483d bytes) 000099[0] 1789124573.909285 write_OD 0x1aa000 thru 0x1acfff (0x3000 bytes) 000100[0] 1789124573.980054 mapread 0x1c7000 thru 0x1d62c3 (0xf2c4 bytes) 000101[1] 1789124574.015304 read_OD 0x057000 thru 0x05ffff (0x9000 bytes) 000103[1] 1789124574.046521 mapread 0x19e000 thru 0x1a4cf9 (0x6cfa bytes) 000105[0] 1789124574.124714 write_OD 0x57c000 thru 0x588fff (0xd000 bytes) 000107[0] 1789124574.218647 read_OD 0x566000 thru 0x574fff (0xf000 bytes) 000108[1] 1789124574.242450 read_OD 0x25d000 thru 0x25ffff (0x3000 bytes) 000109[1] 1789124574.248024 trunc from 0x589000 to 0x0e6b0e (0x4a24f1 bytes) 000111[1] 1789124574.308409 read_OD 0x031000 thru 0x039fff (0x9000 bytes) 000113[0] 1789124574.333055 trunc from 0x0e6b0f to 0x30a50b (0x2239fd bytes) 000118[0] 1789124574.438654 trunc from 0x30a50c to 0x32c523 (0x22018 bytes) 000119[1] 1789124574.476420 trunc from 0x32c524 to 0x6b1fc0 (0x385a9d bytes) 000125[0] 1789124574.556298 trunc from 0x6b1fc1 to 0x1332c3 (0x57ecfd bytes) 000126[0] 1789124574.611886 mapwrite 0x4e0000 thru 0x4e2a32 (0x2a33 bytes) 000127[0] 1789124574.756078 read_OD 0x251000 thru 0x252fff (0x2000 bytes) 000128[1] 1789124574.773723 read_OD 0x39e000 thru 0x39ffff (0x2000 bytes) 000129[0] 1789124574.802095 mapwrite 0x309000 thru 0x316b23 (0xdb24 bytes) 000132[1] 1789124574.967513 mapwrite 0x8a4000 thru 0x8b2c87 (0xec88 bytes) 000133[1] 1789124575.127331 write_OD 0x03a000 thru 0x043fff (0xa000 bytes) 000137[0] 1789124575.226569 mapwrite 0x81d000 thru 0x828ec6 (0xbec7 bytes) 000138[0] 1789124575.391294 mapwrite 0x21d000 thru 0x225f0b (0x8f0c bytes) 000139[0] 1789124575.490342 mapread 0x6ae000 thru 0x6bd271 (0xf272 bytes) 000140[1] 1789124575.528531 trunc from 0x8b2c88 to 0x829508 (0x8977f bytes) 000142[1] 1789124575.610127 mapread 0x051000 thru 0x0557ea (0x47eb bytes) 000144[1] 1789124575.639750 write_OD 0x970000 thru 0x977fff (0x8000 bytes) 000146[0] 1789124575.714676 mapread 0x44d000 thru 0x458f10 (0xbf11 bytes) 000150[0] 1789124575.781799 write_OD 0x8ab000 thru 0x8b1fff (0x7000 bytes) 000152[1] 1789124575.841118 mapwrite 0x963000 thru 0x9635d3 (0x5d4 bytes) 000155[1] 1789124575.963765 trunc from 0x978000 to 0x426ffd (0x551002 bytes) 000156[0] 1789124576.007033 read_OD 0x202000 thru 0x20cfff (0xb000 bytes) 000158[1] 1789124576.064075 write_OD 0x6af000 thru 0x6b5fff (0x7000 bytes) 000160[1] 1789124576.128980 mapread 0x17c000 thru 0x18193b (0x593c bytes) 000164[1] 1789124576.149420 trunc from 0x6b6000 to 0x41508d (0x2a0f72 bytes) 000165[1] 1789124576.209110 trunc from 0x41508e to 0x991bf4 (0x57cb67 bytes) 000167[0] 1789124576.273143 mapwrite 0x227000 thru 0x22b614 (0x4615 bytes) 000168[1] 1789124576.416437 mapread 0x1ef000 thru 0x1f969a (0xa69b bytes) 000171[0] 1789124576.475442 mapread 0x96b000 thru 0x96ece3 (0x3ce4 bytes) 000173[1] 1789124576.512005 trunc from 0x991bf5 to 0x0ac318 (0x8e58dc bytes) 000174[1] 1789124576.577941 mapwrite 0x2c8000 thru 0x2d638d (0xe38e bytes) 000176[0] 1789124576.779132 read_OD 0x088000 thru 0x091fff (0xa000 bytes) 000177[1] 1789124576.825060 mapread 0x1d9000 thru 0x1dccc1 (0x3cc2 bytes) 000179[1] 1789124576.849790 trunc from 0x2d638e to 0x20ab77 (0xcb816 bytes) 000180[0] 1789124576.920728 mapread 0x19d000 thru 0x1ab50a (0xe50b bytes) 000184[1] 1789124577.001498 mapwrite 0x0b5000 thru 0x0b7645 (0x2646 bytes) 000185[1] 1789124577.108724 write_OD 0x3f7000 thru 0x3fffff (0x9000 bytes) 000187[0] 1789124577.156279 trunc from 0x400000 to 0x916c49 (0x516c4a bytes) 000188[1] 1789124577.205576 mapwrite 0x38f000 thru 0x39e928 (0xf929 bytes) 000189[1] 1789124577.371725 trunc from 0x916c4a to 0x9a933c (0x926f3 bytes) 000190[1] 1789124577.397961 write_OD 0x968000 thru 0x971fff (0xa000 bytes) 000192[0] 1789124577.518501 mapread 0x908000 thru 0x9166fb (0xe6fc bytes) 000194[1] 1789124577.660521 mapwrite 0x28c000 thru 0x2916e1 (0x56e2 bytes) 000197[0] 1789124577.833201 mapread 0x133000 thru 0x13a392 (0x7393 bytes) 000198[0] 1789124577.892387 trunc from 0x9a933d to 0x7fdbab (0x1ab791 bytes) 000199[0] 1789124577.942865 read_OD 0x650000 thru 0x654fff (0x5000 bytes) 000201[1] 1789124577.984870 read_OD 0x067000 thru 0x072fff (0xc000 bytes) 000203[1] 1789124578.004018 write_OD 0x705000 thru 0x70cfff (0x8000 bytes) 000204[0] 1789124578.071581 mapread 0x221000 thru 0x228a80 (0x7a81 bytes) 000207[1] 1789124578.114588 mapwrite 0x2ed000 thru 0x2f7921 (0xa922 bytes) 000212[1] 1789124578.301059 mapread 0x025000 thru 0x030a47 (0xba48 bytes) 000214[1] 1789124578.355799 read_OD 0x46f000 thru 0x479fff (0xb000 bytes) 000217[0] 1789124578.390294 trunc from 0x7fdbac to 0x35214d (0x4aba5e bytes) 000218[0] 1789124578.441072 read_OD 0x28c000 thru 0x28ffff (0x4000 bytes) 000220[0] 1789124578.466105 write_OD 0x267000 thru 0x269fff (0x3000 bytes) 000223[0] 1789124578.552809 mapread 0x257000 thru 0x25cbf7 (0x5bf8 bytes) 000225[1] 1789124578.603706 mapwrite 0x65d000 thru 0x663a32 (0x6a33 bytes) 000226[1] 1789124578.735736 write_OD 0x98b000 thru 0x98cfff (0x2000 bytes) 000228[1] 1789124578.798496 mapread 0x7c6000 thru 0x7c7bb9 (0x1bba bytes) 000232[1] 1789124578.819055 mapwrite 0x960000 thru 0x960f15 (0xf16 bytes) 000234[0] 1789124578.913392 write_OD 0x6f2000 thru 0x6f7fff (0x6000 bytes) 000236[0] 1789124579.003208 trunc from 0x98d000 to 0x40c339 (0x580cc6 bytes) 000240[1] 1789124579.122345 read_OD 0x1c1000 thru 0x1cefff (0xe000 bytes) 000241[0] 1789124579.159077 read_OD 0x092000 thru 0x0a0fff (0xf000 bytes) 000245[0] 1789124579.180474 trunc from 0x40c33a to 0x4c8ee7 (0xbcbae bytes) 000247[0] 1789124579.282429 mapwrite 0x930000 thru 0x9308b1 (0x8b2 bytes) 000248[1] 1789124579.390170 mapread 0x6fe000 thru 0x7041b4 (0x61b5 bytes) 000253[1] 1789124579.443420 trunc from 0x9308b2 to 0x0f16bd (0x83f1f4 bytes) 000258[0] 1789124579.554896 read_OD 0x05b000 thru 0x061fff (0x7000 bytes) 000262[1] 1789124579.589208 mapread 0x0a3000 thru 0x0a83ef (0x53f0 bytes) 000264[1] 1789124579.616949 mapread 0x02c000 thru 0x035a53 (0x9a54 bytes) 000268[1] 1789124579.636797 trunc from 0x0f16be to 0x849b87 (0x7584ca bytes) 000269[1] 1789124579.729891 write_OD 0x3c9000 thru 0x3cbfff (0x3000 bytes) 000271[1] 1789124579.857593 write_OD 0x002000 thru 0x00ffff (0xe000 bytes) 000272[0] 1789124579.936081 mapwrite 0x299000 thru 0x29cc2e (0x3c2f bytes) 000273[0] 1789124580.173007 write_OD 0x364000 thru 0x36ffff (0xc000 bytes) 000274[1] 1789124580.317585 trunc from 0x849b88 to 0x431cf5 (0x417e92 bytes) 000275[1] 1789124580.424084 mapwrite 0x34e000 thru 0x357821 (0x9822 bytes) 000276[0] 1789124580.568717 mapread 0x3d8000 thru 0x3dff41 (0x7f42 bytes) 000279[0] 1789124580.655765 mapwrite 0x5e9000 thru 0x5f84e9 (0xf4ea bytes) 000281[0] 1789124580.851443 write_OD 0x58d000 thru 0x598fff (0xc000 bytes) 000284[1] 1789124580.937874 read_OD 0x403000 thru 0x40dfff (0xb000 bytes) 000287[1] 1789124580.950645 mapwrite 0x08a000 thru 0x08e2b6 (0x42b7 bytes) 000288[0] 1789124581.036368 read_OD 0x254000 thru 0x25bfff (0x8000 bytes) 000290[0] 1789124581.095876 trunc from 0x5f84ea to 0x6f59e8 (0xfd4ff bytes) 000294[1] 1789124581.180058 read_OD 0x63b000 thru 0x640fff (0x6000 bytes) 000295[0] 1789124581.204987 trunc from 0x6f59e9 to 0x192372 (0x563676 bytes) 000299[1] 1789124581.330864 read_OD 0x005000 thru 0x013fff (0xf000 bytes) 000300[1] 1789124581.384727 trunc from 0x192373 to 0x4cb2b9 (0x338f47 bytes) 000302[1] 1789124581.433883 trunc from 0x4cb2ba to 0x29bdb6 (0x22f503 bytes) 000304[1] 1789124581.474932 trunc from 0x29bdb7 to 0x7ec027 (0x550271 bytes) 000305[0] 1789124581.535871 write_OD 0x669000 thru 0x674fff (0xc000 bytes) 000306[0] 1789124581.630071 write_OD 0x115000 thru 0x118fff (0x4000 bytes) 000307[0] 1789124581.710352 mapread 0x0b1000 thru 0x0be472 (0xd473 bytes) 000308[1] 1789124581.779732 trunc from 0x7ec028 to 0x14422d (0x6a7dfa bytes) 000314[1] 1789124581.886844 mapread 0x00e000 thru 0x00edb5 (0xdb6 bytes) 000319[0] 1789124582.018770 mapwrite 0x267000 thru 0x2671c4 (0x1c5 bytes) 000320[0] 1789124582.100180 mapwrite 0x4e2000 thru 0x4e58b7 (0x38b8 bytes) 000321[0] 1789124582.191868 write_OD 0x400000 thru 0x400fff (0x1000 bytes) 000323[1] 1789124582.251694 mapwrite 0x55c000 thru 0x5678e3 (0xb8e4 bytes) 000324[0] 1789124582.413389 mapread 0x0a3000 thru 0x0b2e56 (0xfe57 bytes) 000326[1] 1789124582.500655 trunc from 0x5678e4 to 0x37eebd (0x1e8a26 bytes) 000327[0] 1789124582.557770 mapwrite 0x961000 thru 0x96d759 (0xc75a bytes) 000328[0] 1789124582.715190 mapwrite 0x9c6000 thru 0x9c8bc8 (0x2bc9 bytes) 000329[1] 1789124582.824732 write_OD 0x385000 thru 0x387fff (0x3000 bytes) 000336[0] 1789124582.917591 mapwrite 0x567000 thru 0x568286 (0x1287 bytes) 000339[1] 1789124583.083174 write_OD 0x8e0000 thru 0x8e5fff (0x6000 bytes) 000340[0] 1789124583.176396 trunc from 0x9c8bc9 to 0x89bf58 (0x12cc70 bytes) 000343[1] 1789124583.262914 trunc from 0x89bf59 to 0x31cbb5 (0x57f3a3 bytes) 000345[1] 1789124583.330835 read_OD 0x309000 thru 0x30afff (0x2000 bytes) 000348[1] 1789124583.352683 write_OD 0x2aa000 thru 0x2b4fff (0xb000 bytes) 000349[1] 1789124583.423824 mapread 0x193000 thru 0x1a1f75 (0xef76 bytes) 000351[1] 1789124583.508743 mapread 0x23a000 thru 0x23c50a (0x250b bytes) 000354[1] 1789124583.510070 mapwrite 0x7bd000 thru 0x7bed11 (0x1d12 bytes) 000355[0] 1789124583.692070 mapwrite 0x3ed000 thru 0x3f7861 (0xa862 bytes) 000357[0] 1789124583.823166 trunc from 0x7bed12 to 0x8fe7e0 (0x13facf bytes) 000358[0] 1789124583.858094 write_OD 0x5a7000 thru 0x5b5fff (0xf000 bytes) 000361[1] 1789124583.933903 read_OD 0x6ae000 thru 0x6b4fff (0x7000 bytes) 000362[0] 1789124583.970018 mapwrite 0x008000 thru 0x00ec88 (0x6c89 bytes) 000363[0] 1789124584.068916 mapread 0x87b000 thru 0x8873bd (0xc3be bytes) 000364[1] 1789124584.131996 write_OD 0x15b000 thru 0x15efff (0x4000 bytes) 000365[0] 1789124584.232287 trunc from 0x8fe7e1 to 0x07ee65 (0x87f97b bytes) 000366[0] 1789124584.275401 read_OD 0x067000 thru 0x071fff (0xb000 bytes) 000367[1] 1789124584.311180 mapwrite 0x9a1000 thru 0x9a86a7 (0x76a8 bytes) 000368[0] 1789124584.466864 write_OD 0x0cc000 thru 0x0cdfff (0x2000 bytes) 000369[0] 1789124584.539857 mapread 0x2e2000 thru 0x2eb7af (0x97b0 bytes) 000371[0] 1789124584.658659 write_OD 0x646000 thru 0x648fff (0x3000 bytes) 000372[0] 1789124584.737451 mapwrite 0x085000 thru 0x092764 (0xd765 bytes) 000378[0] 1789124584.913020 trunc from 0x9a86a8 to 0x938acf (0x6fbd8 bytes) 000381[0] 1789124584.934989 mapwrite 0x14e000 thru 0x154486 (0x6487 bytes) 000382[0] 1789124585.057611 mapwrite 0x82c000 thru 0x838b25 (0xcb26 bytes) 000384[1] 1789124585.228100 mapwrite 0x87c000 thru 0x881271 (0x5272 bytes) 000385[1] 1789124585.353142 mapwrite 0x0ba000 thru 0x0ba460 (0x461 bytes) 000386[0] 1789124585.405483 mapread 0x501000 thru 0x506d72 (0x5d73 bytes) 000387[1] 1789124585.465051 read_OD 0x038000 thru 0x03afff (0x3000 bytes) 000388[0] 1789124585.471696 read_OD 0x887000 thru 0x892fff (0xc000 bytes) 000389[1] 1789124585.479713 mapread 0x547000 thru 0x55569e (0xe69f bytes) 000400[0] 1789124585.523496 mapwrite 0x5bd000 thru 0x5c23c1 (0x53c2 bytes) 000402[1] 1789124585.647602 read_OD 0x7f7000 thru 0x803fff (0xd000 bytes) 000403[1] 1789124585.701805 mapwrite 0x197000 thru 0x1a5835 (0xe836 bytes) 000405[1] 1789124585.820607 mapread 0x805000 thru 0x80c1c6 (0x71c7 bytes) 000406[0] 1789124585.861021 trunc from 0x938ad0 to 0x076b8c (0x8c1f43 bytes) 000409[1] 1789124585.899468 mapread 0x049000 thru 0x04a1da (0x11db bytes) 000411[1] 1789124585.941525 write_OD 0x904000 thru 0x906fff (0x3000 bytes) 000414[0] 1789124586.012718 read 0x079000 thru 0x07b4e8 (0x24e9 bytes) 000415[1] 1789124586.019486 trunc from 0x907000 to 0x887ce0 (0x7f31f bytes) 000418[1] 1789124586.101410 mapread 0x14e000 thru 0x159116 (0xb117 bytes) 000419[1] 1789124586.133571 trunc from 0x887ce1 to 0x98ec23 (0x106f43 bytes) 000422[1] 1789124586.214667 write_OD 0x9d9000 thru 0x9e7fff (0xf000 bytes) 000423[0] 1789124586.360417 trunc from 0x9e8000 to 0x624ef8 (0x3c3107 bytes) 000428[0] 1789124586.459348 mapwrite 0x367000 thru 0x36bd2f (0x4d30 bytes) 000430[0] 1789124586.539409 trunc from 0x624ef9 to 0x65f622 (0x3a72a bytes) 000431[0] 1789124586.577797 mapread 0x5fa000 thru 0x5fa13a (0x13b bytes) 000433[0] 1789124586.615141 mapread 0x405000 thru 0x40c7a8 (0x77a9 bytes) 000437[0] 1789124586.707449 mapwrite 0x9eb000 thru 0x9f5982 (0xa983 bytes) 000439[0] 1789124586.871177 write 0x40d000 thru 0x41b0e1 (0xe0e2 bytes) 000441[1] 1789124586.942335 mapwrite 0x635000 thru 0x643aea (0xeaeb bytes) 000442[0] 1789124587.204004 trunc from 0x9f5983 to 0x6c426a (0x331718 bytes) 000443[1] 1789124587.258671 mapwrite 0x5d6000 thru 0x5e088f (0xa890 bytes) 000444[1] 1789124587.368685 mapread 0x075000 thru 0x080320 (0xb321 bytes) 000445[1] 1789124587.407759 mapread 0x18b000 thru 0x18b2af (0x2b0 bytes) 000447[1] 1789124587.412836 trunc from 0x6c426b to 0x1dd2b0 (0x4e6fba bytes) 000448[1] 1789124587.432281 trunc from 0x1dd2b1 to 0x22f34b (0x5209b bytes) 000450[0] 1789124587.469023 mapread 0x145000 thru 0x14ce0c (0x7e0d bytes) 000452[1] 1789124587.514340 mapread 0x0c8000 thru 0x0d53cd (0xd3ce bytes) 000453[0] 1789124587.544408 read 0x069000 thru 0x06cb32 (0x3b33 bytes) 000454[0] 1789124587.586925 read 0x07f000 thru 0x08e1be (0xf1bf bytes) 000457[0] 1789124587.596976 read 0x175000 thru 0x176e08 (0x1e09 bytes) 000458[0] 1789124587.599641 read 0x096000 thru 0x0a0979 (0xa97a bytes) 000460[0] 1789124587.600386 mapwrite 0x9c5000 thru 0x9c9fc4 (0x4fc5 bytes) 000462[0] 1789124588.043886 read 0x450000 thru 0x45c810 (0xc811 bytes) 000466[1] 1789124588.106718 mapread 0x2e3000 thru 0x2e6bc2 (0x3bc3 bytes) truncating to largest ever: 0x9c1b0a 000467[0] 1789124588.185894 trunc from 0x9c9fc5 to 0x9c1b09 (0x84bb bytes) 000470[1] 1789124588.256098 mapwrite 0x6a5000 thru 0x6a9641 (0x4642 bytes) 000472[1] 1789124588.359741 read_OD 0x215000 thru 0x219fff (0x5000 bytes) 000474[1] 1789124588.379635 write_OD 0x45f000 thru 0x462fff (0x4000 bytes) 000475[0] 1789124588.432680 write 0x433000 thru 0x44238f (0xf390 bytes) 000476[0] 1789124588.460620 write 0x727000 thru 0x727744 (0x745 bytes) 000477[1] 1789124588.508360 read_OD 0x622000 thru 0x627fff (0x6000 bytes) 000479[0] 1789124588.586490 mapwrite 0x8f2000 thru 0x8f860f (0x6610 bytes) 000480[0] 1789124588.667616 mapread 0x0e0000 thru 0x0e9b6b (0x9b6c bytes) 000481[0] 1789124588.701091 mapwrite 0x3e7000 thru 0x3ecda1 (0x5da2 bytes) 000482[1] 1789124588.819785 read_OD 0x969000 thru 0x977fff (0xf000 bytes) 000484[1] 1789124588.861432 read_OD 0x6ce000 thru 0x6d8fff (0xb000 bytes) 000485[1] 1789124588.872719 write_OD 0x64a000 thru 0x656fff (0xd000 bytes) 000488[1] 1789124588.934409 mapread 0x1dd000 thru 0x1e9137 (0xc138 bytes) 000490[0] 1789124588.972783 mapread 0x125000 thru 0x125411 (0x412 bytes) 000491[0] 1789124588.977942 read 0x7d7000 thru 0x7e3a05 (0xca06 bytes) 000492[1] 1789124588.988030 mapwrite 0x421000 thru 0x42f4ec (0xe4ed bytes) 000493[1] 1789124589.065732 write_OD 0x87d000 thru 0x881fff (0x5000 bytes) 000497[0] 1789124589.132680 read 0x0ba000 thru 0x0c656d (0xc56e bytes) 000499[1] 1789124589.141832 write_OD 0x0ac000 thru 0x0b8fff (0xd000 bytes) All operations completed A-OK! PASS 16a (53s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 07:03:13 (1789124593) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.380195 s, 22.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.317822 s, 26.4 MB/s Chance of close/open is 1 in 50 Seed set to 4596 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x54c93e 000001[1] 1789124596.754706 trunc from 00000000 to 0x54c93d (0x54c93e bytes) 000002[0] 1789124596.834005 mapread 0x3727ba thru 0x37476a (0x1fb1 bytes) 000006[0] 1789124596.969304 mapwrite 0x2cf1a0 thru 0x2d05dc (0x143d bytes) 000007[1] 1789124597.090885 write 0x6de283 thru 0x6df117 (0xe95 bytes) 000009[1] 1789124597.128834 mapwrite 0x8ced9a thru 0x8d0a8a (0x1cf1 bytes) 000010[1] 1789124597.359570 mapread 0x4d261b thru 0x4d2ab6 (0x49c bytes) 000013[1] 1789124597.413890 read 0x6bd898 thru 0x6be66d (0xdd6 bytes) 000017[0] 1789124597.495841 mapread 0x413659 thru 0x4152be (0x1c66 bytes) truncating to largest ever: 0x88773e 000019[1] 1789124597.587785 trunc from 0x8d0a8b to 0x88773d (0x4934d bytes) 000022[1] 1789124597.693397 mapread 0x565232 thru 0x565eab (0xc7a bytes) 000023[0] 1789124597.719547 write 0x6e08b5 thru 0x6e26f3 (0x1e3f bytes) 000024[1] 1789124597.757852 mapread 0x75f709 thru 0x7612db (0x1bd3 bytes) 000025[1] 1789124597.818643 write 0x3f271d thru 0x3f2802 (0xe6 bytes) 000030[1] 1789124597.856528 mapread 0x859d04 thru 0x85b3ba (0x16b7 bytes) 000031[0] 1789124597.867930 write 0x1ddf9b thru 0x1df98f (0x19f5 bytes) 000032[0] 1789124597.893044 mapread 0x51942f thru 0x519f11 (0xae3 bytes) 000033[0] 1789124597.912145 write 0x3e361d thru 0x3e3d48 (0x72c bytes) 000035[1] 1789124597.933846 read 0x35366a thru 0x355524 (0x1ebb bytes) 000036[0] 1789124597.954017 mapwrite 0x318de9 thru 0x319e00 (0x1018 bytes) 000037[1] 1789124598.018968 write 0x60e72f thru 0x6105f8 (0x1eca bytes) 000038[1] 1789124598.059526 write 0x91b632 thru 0x91c883 (0x1252 bytes) 000039[0] 1789124598.098723 mapwrite 0x922cea thru 0x9246e6 (0x19fd bytes) 000044[1] 1789124598.295699 mapread 0x0771c3 thru 0x07847b (0x12b9 bytes) truncating to largest ever: 0x949b97 000049[0] 1789124598.437150 trunc from 0x9246e7 to 0x949b96 (0x254b0 bytes) 000050[0] 1789124598.489948 mapwrite 0x194806 thru 0x196664 (0x1e5f bytes) 000058[0] 1789124598.706859 write 0x6a36b1 thru 0x6a4fec (0x193c bytes) 000060[1] 1789124598.771156 trunc from 0x949b97 to 0x4c9441 (0x480755 bytes) 000062[0] 1789124598.860666 write 0x0d9d39 thru 0x0daaa4 (0xd6c bytes) 000064[1] 1789124598.930093 write 0x5ef36f thru 0x5ef9e0 (0x672 bytes) 000065[1] 1789124598.957143 mapread 0x3f557e thru 0x3f69fa (0x147d bytes) 000068[1] 1789124599.047573 trunc from 0x5ef9e1 to 0x0a55b4 (0x54a42c bytes) 000069[0] 1789124599.167353 mapread 0x06157a thru 0x0621c4 (0xc4b bytes) 000072[0] 1789124599.299105 read 0x0269d1 thru 0x026b8f (0x1bf bytes) 000074[0] 1789124599.306467 write 0x603f6d thru 0x60565f (0x16f3 bytes) 000076[1] 1789124599.367731 mapread 0x580372 thru 0x580d34 (0x9c3 bytes) 000077[0] 1789124599.428923 trunc from 0x605660 to 0x230dcc (0x3d4893 bytes) 000078[1] 1789124599.519069 mapwrite 0x3699a8 thru 0x36a5e3 (0xc3c bytes) 000079[0] 1789124599.741400 read 0x090442 thru 0x09064e (0x20d bytes) 000080[0] 1789124599.835723 read 0x255ad2 thru 0x256573 (0xaa2 bytes) 000085[0] 1789124599.869147 trunc from 0x36a5e4 to 0x8aa562 (0x53ff7f bytes) 000086[1] 1789124600.001970 write 0x3252b6 thru 0x326f55 (0x1ca0 bytes) 000095[0] 1789124600.206079 mapread 0x0f3c8c thru 0x0f4957 (0xccc bytes) 000097[1] 1789124600.320889 write 0x840f36 thru 0x8412e0 (0x3ab bytes) 000098[1] 1789124600.384288 write 0x6794a3 thru 0x67b43a (0x1f98 bytes) 000100[1] 1789124600.451510 write 0x7acebb thru 0x7ad97e (0xac4 bytes) 000101[1] 1789124600.467783 mapread 0x46c9c3 thru 0x46cf01 (0x53f bytes) 000103[0] 1789124600.494936 mapread 0x2b6676 thru 0x2b735a (0xce5 bytes) truncating to largest ever: 0x97ad4c 000105[0] 1789124600.593134 trunc from 0x8aa563 to 0x97ad4b (0xd07e9 bytes) 000107[0] 1789124600.688930 mapread 0x956a24 thru 0x956b05 (0xe2 bytes) 000108[1] 1789124600.733238 trunc from 0x97ad4c to 0x275ab9 (0x705292 bytes) 000109[0] 1789124600.832135 mapread 0x227c14 thru 0x2291bd (0x15aa bytes) 000110[0] 1789124600.894389 mapread 0x2453e0 thru 0x247267 (0x1e88 bytes) 000111[1] 1789124600.951889 read 0x1a5521 thru 0x1a57b8 (0x298 bytes) 000115[0] 1789124601.066816 mapwrite 0x5303b7 thru 0x53141b (0x1065 bytes) 000117[1] 1789124601.221359 read 0x1fcc50 thru 0x1fdbb0 (0xf61 bytes) 000118[0] 1789124601.372129 mapread 0x183677 thru 0x185378 (0x1d02 bytes) 000119[1] 1789124601.416609 write 0x3b4e98 thru 0x3b63e1 (0x154a bytes) 000121[0] 1789124601.526059 mapread 0x36895a thru 0x36a25c (0x1903 bytes) 000124[1] 1789124601.602672 read 0x4e3447 thru 0x4e39e9 (0x5a3 bytes) 000128[1] 1789124601.629497 mapread 0x3220a4 thru 0x323007 (0xf64 bytes) 000129[1] 1789124601.643970 trunc from 0x53141c to 0x30f98c (0x221a8f bytes) 000130[0] 1789124601.724325 mapread 0x1bba61 thru 0x1bd33f (0x18df bytes) 000131[1] 1789124601.792653 write 0x7d3b48 thru 0x7d53a0 (0x1859 bytes) 000134[0] 1789124601.830619 write 0x697eeb thru 0x69963c (0x1752 bytes) 000135[0] 1789124601.859041 mapread 0x4984fd thru 0x498696 (0x19a bytes) 000136[0] 1789124601.889302 write 0x17901c thru 0x179758 (0x73d bytes) 000137[0] 1789124601.911952 read 0x13d5a8 thru 0x13dc22 (0x67b bytes) 000138[1] 1789124601.917811 mapwrite 0x76c19a thru 0x76d668 (0x14cf bytes) 000140[1] 1789124602.100359 read 0x30c0ad thru 0x30c95a (0x8ae bytes) 000142[1] 1789124602.162720 mapwrite 0x5564fa thru 0x5583b0 (0x1eb7 bytes) 000145[0] 1789124602.360458 trunc from 0x7d53a1 to 0x32bbc4 (0x4a97dc bytes) 000146[1] 1789124602.408066 trunc from 0x32bbc5 to 0x3253a9 (0x681b bytes) 000147[0] 1789124602.451849 trunc from 0x3253aa to 0x3364f1 (0x11148 bytes) 000148[0] 1789124602.492973 write 0x744a83 thru 0x744b05 (0x83 bytes) 000151[1] 1789124602.575703 mapwrite 0x7372b8 thru 0x7380ba (0xe03 bytes) truncating to largest ever: 0x9ec20b 000153[0] 1789124602.815500 trunc from 0x744b06 to 0x9ec20a (0x2a7705 bytes) 000155[0] 1789124602.888732 mapwrite 0x11311b thru 0x113676 (0x55c bytes) 000156[1] 1789124602.983405 mapread 0x065c40 thru 0x067bda (0x1f9b bytes) 000157[0] 1789124603.061981 mapread 0x3117b8 thru 0x313430 (0x1c79 bytes) 000160[1] 1789124603.100978 write 0x8a501a thru 0x8a53cf (0x3b6 bytes) 000162[0] 1789124603.152604 read 0x2d9a62 thru 0x2da809 (0xda8 bytes) 000164[0] 1789124603.207435 read 0x485d7d thru 0x485fe2 (0x266 bytes) 000165[1] 1789124603.230477 trunc from 0x9ec20b to 0x9915d1 (0x5ac39 bytes) 000166[0] 1789124603.288286 write 0x2cbe65 thru 0x2cdd77 (0x1f13 bytes) 000167[1] 1789124603.350496 read 0x1cfbb0 thru 0x1d17f2 (0x1c43 bytes) 000169[0] 1789124603.381504 trunc from 0x9915d2 to 0x1e5a72 (0x7abb5f bytes) 000171[0] 1789124603.431892 read 0x0ccca8 thru 0x0cd7ab (0xb04 bytes) 000172[0] 1789124603.468848 mapread 0x01f00c thru 0x020b56 (0x1b4b bytes) 000173[0] 1789124603.470788 read 0x0729df thru 0x072cfd (0x31f bytes) 000174[0] 1789124603.497135 read 0x14193f thru 0x141a7e (0x140 bytes) 000177[1] 1789124603.499582 read 0x1acbc2 thru 0x1ad5f5 (0xa34 bytes) 000178[1] 1789124603.561843 trunc from 0x1e5a73 to 0x0b4259 (0x131819 bytes) 000180[1] 1789124603.661977 write 0x5b29dd thru 0x5b3c5c (0x1280 bytes) 000182[1] 1789124603.696920 read 0x4615c4 thru 0x462bbf (0x15fc bytes) 000184[0] 1789124603.722273 write 0x472679 thru 0x474551 (0x1ed9 bytes) 000190[0] 1789124603.872838 write 0x2d07b2 thru 0x2d1e2f (0x167e bytes) 000192[1] 1789124603.937103 trunc from 0x5b3c5d to 0x470aa0 (0x1431bc bytes) 000193[1] 1789124603.991300 mapwrite 0x21a913 thru 0x21c2a9 (0x1997 bytes) 000194[0] 1789124604.050851 trunc from 0x470aa1 to 0x88dcc2 (0x41d222 bytes) 000195[0] 1789124604.125845 read 0x813a22 thru 0x81413f (0x71e bytes) 000196[1] 1789124604.157639 read 0x0e7266 thru 0x0e81e4 (0xf7f bytes) 000197[0] 1789124604.185427 mapread 0x5d0847 thru 0x5d0ea9 (0x663 bytes) 000200[0] 1789124604.206039 mapread 0x4f2a4d thru 0x4f3a6f (0x1023 bytes) 000203[1] 1789124604.275427 read 0x72594b thru 0x7260f8 (0x7ae bytes) 000204[1] 1789124604.308846 mapread 0x232a06 thru 0x233f42 (0x153d bytes) 000207[0] 1789124604.347973 trunc from 0x88dcc3 to 0x8d2301 (0x4463f bytes) 000208[1] 1789124604.405636 read 0x0d5506 thru 0x0d74e3 (0x1fde bytes) 000213[1] 1789124604.476953 read 0x655d46 thru 0x657210 (0x14cb bytes) 000216[0] 1789124604.506817 trunc from 0x8d2302 to 0x2b394c (0x61e9b5 bytes) 000217[0] 1789124604.597876 write 0x342475 thru 0x343451 (0xfdd bytes) 000220[0] 1789124604.642801 mapread 0x2a24a6 thru 0x2a4146 (0x1ca1 bytes) 000221[0] 1789124604.687618 write 0x5896d9 thru 0x589899 (0x1c1 bytes) 000222[1] 1789124604.690750 mapread 0x52a2a2 thru 0x52b134 (0xe93 bytes) 000223[1] 1789124604.749495 read 0x17b07f thru 0x17c4bf (0x1441 bytes) 000224[0] 1789124604.783302 read 0x4f547f thru 0x4f62a3 (0xe25 bytes) 000225[1] 1789124604.792711 write 0x4966de thru 0x496949 (0x26c bytes) 000228[1] 1789124604.855400 mapwrite 0x560c52 thru 0x561f55 (0x1304 bytes) 000230[1] 1789124605.018438 write 0x33d827 thru 0x33ec9a (0x1474 bytes) 000232[1] 1789124605.122846 write 0x524b0b thru 0x5250a0 (0x596 bytes) 000236[0] 1789124605.256781 mapread 0x30cdab thru 0x30e414 (0x166a bytes) 000239[1] 1789124605.468992 mapwrite 0x367637 thru 0x3685f4 (0xfbe bytes) 000240[1] 1789124605.662340 read 0x02c3f0 thru 0x02cde0 (0x9f1 bytes) 000241[1] 1789124605.670650 write 0x8da787 thru 0x8db8c9 (0x1143 bytes) 000243[0] 1789124605.766466 write 0x42406e thru 0x425c36 (0x1bc9 bytes) 000244[0] 1789124605.877848 mapread 0x5b0d41 thru 0x5b11d3 (0x493 bytes) 000247[1] 1789124605.995618 mapwrite 0x9bd782 thru 0x9be8f8 (0x1177 bytes) 000248[1] 1789124606.172798 read 0x65375f thru 0x6548c9 (0x116b bytes) 000250[1] 1789124606.212344 write 0x2914fa thru 0x292da6 (0x18ad bytes) 000254[1] 1789124606.258733 mapwrite 0x38ebcb thru 0x38fcb5 (0x10eb bytes) 000257[1] 1789124606.393904 mapread 0x0d100f thru 0x0d1da3 (0xd95 bytes) 000259[1] 1789124606.428963 trunc from 0x9be8f9 to 0x8b66ae (0x10824a bytes) 000260[0] 1789124606.460765 trunc from 0x8b66af to 0x1ae79c (0x707f12 bytes) 000261[1] 1789124606.539773 read 0x02965d thru 0x02a2bc (0xc60 bytes) 000263[0] 1789124606.628044 mapread 0x096aa5 thru 0x097590 (0xaec bytes) 000265[1] 1789124606.638550 write 0x0f8bce thru 0x0f954c (0x97f bytes) 000266[1] 1789124606.707388 mapread 0x05bdc6 thru 0x05d77c (0x19b7 bytes) 000268[0] 1789124606.871943 read 0x05868b thru 0x058847 (0x1bd bytes) 000269[1] 1789124606.970358 read 0x158578 thru 0x159e58 (0x18e1 bytes) 000271[0] 1789124607.083937 read 0x0dcfa1 thru 0x0dd9a5 (0xa05 bytes) 000272[1] 1789124607.084898 mapwrite 0x9be538 thru 0x9bf3c3 (0xe8c bytes) 000278[0] 1789124607.480740 mapread 0x40b173 thru 0x40c680 (0x150e bytes) 000279[0] 1789124607.531814 read 0x15339e thru 0x154156 (0xdb9 bytes) 000282[1] 1789124607.587583 trunc from 0x9bf3c4 to 0x1e6eb0 (0x7d8513 bytes) 000283[1] 1789124607.681272 mapread 0x13ba91 thru 0x13c010 (0x580 bytes) 000285[1] 1789124607.783991 read 0x00daef thru 0x00f3d0 (0x18e2 bytes) 000286[1] 1789124607.870863 mapwrite 0x161d6b thru 0x161fbd (0x253 bytes) 000288[0] 1789124607.984865 write 0x0a0319 thru 0x0a0d71 (0xa59 bytes) 000292[1] 1789124608.171675 mapread 0x125e78 thru 0x126e1e (0xfa7 bytes) 000294[1] 1789124608.245158 mapwrite 0x6d91b6 thru 0x6da3dd (0x1228 bytes) 000295[1] 1789124608.514951 write 0x4db6a3 thru 0x4dc086 (0x9e4 bytes) 000296[0] 1789124608.536339 read 0x35b629 thru 0x35ba79 (0x451 bytes) 000297[0] 1789124608.578923 trunc from 0x6da3de to 0x9205a1 (0x2461c4 bytes) 000298[0] 1789124608.692960 read 0x264bc6 thru 0x265bd3 (0x100e bytes) 000300[0] 1789124608.768792 trunc from 0x9205a2 to 0x254aa3 (0x6cbafe bytes) 000302[1] 1789124608.865961 read 0x1b28af thru 0x1b4081 (0x17d3 bytes) 000308[0] 1789124609.014927 read 0x045d31 thru 0x0471d8 (0x14a8 bytes) 000312[1] 1789124609.155743 write 0x582e00 thru 0x58349f (0x6a0 bytes) 000314[0] 1789124609.234959 trunc from 0x5834a0 to 0x50c68e (0x76e11 bytes) 000315[0] 1789124609.308206 mapread 0x037362 thru 0x03911d (0x1dbc bytes) 000318[1] 1789124609.466339 write 0x173a08 thru 0x173ea8 (0x4a1 bytes) 000319[0] 1789124609.615567 trunc from 0x50c68f to 0x725a36 (0x2193a8 bytes) 000320[1] 1789124609.722683 read 0x69efcb thru 0x69fbb5 (0xbeb bytes) 000321[1] 1789124609.769741 write 0x4ff175 thru 0x5004cc (0x1358 bytes) 000326[1] 1789124610.028259 mapwrite 0x5cc8ce thru 0x5cd3df (0xb12 bytes) 000328[0] 1789124610.216882 mapwrite 0x84c7ee thru 0x84c820 (0x33 bytes) 000329[1] 1789124610.417866 read 0x2949b1 thru 0x2960a0 (0x16f0 bytes) 000330[0] 1789124610.455545 read 0x72a0fc thru 0x72a6f8 (0x5fd bytes) 000331[0] 1789124610.460070 mapread 0x01670d thru 0x016a74 (0x368 bytes) 000333[1] 1789124610.471340 mapread 0x796132 thru 0x797d4f (0x1c1e bytes) 000339[0] 1789124610.519159 mapread 0x418e0a thru 0x419ccc (0xec3 bytes) 000340[0] 1789124610.529885 mapwrite 0x8ff479 thru 0x90067b (0x1203 bytes) 000341[1] 1789124610.679489 trunc from 0x90067c to 0x52c005 (0x3d4676 bytes) 000342[0] 1789124610.727035 mapwrite 0x5929d5 thru 0x594282 (0x18ae bytes) 000344[0] 1789124610.946342 trunc from 0x594283 to 0x0d890f (0x4bb973 bytes) 000346[0] 1789124611.023628 mapread 0x099f29 thru 0x099f3f (0x17 bytes) 000347[0] 1789124611.054092 mapwrite 0x66f85b thru 0x6705b2 (0xd58 bytes) 000352[1] 1789124611.359301 trunc from 0x6705b3 to 0x863dcf (0x1f381d bytes) 000354[0] 1789124611.475402 mapread 0x61f805 thru 0x6204f9 (0xcf5 bytes) 000356[1] 1789124611.557949 write 0x66c60b thru 0x66dadf (0x14d5 bytes) 000357[1] 1789124611.655683 mapwrite 0x2f6b19 thru 0x2f74c9 (0x9b1 bytes) 000358[1] 1789124611.789071 mapread 0x81100b thru 0x8121a4 (0x119a bytes) 000359[0] 1789124611.806034 mapwrite 0x92d0ac thru 0x92e3ad (0x1302 bytes) 000362[0] 1789124612.058620 mapwrite 0x84b072 thru 0x84c93a (0x18c9 bytes) 000364[0] 1789124612.205520 read 0x788e19 thru 0x78959a (0x782 bytes) 000367[0] 1789124612.378002 trunc from 0x92e3ae to 0x075bef (0x8b87be bytes) 000369[0] 1789124612.530667 trunc from 0x075bf0 to 0x74f639 (0x6d9a4a bytes) 000371[0] 1789124612.715784 read 0x51d74d thru 0x51f4b7 (0x1d6b bytes) 000373[1] 1789124612.804474 read 0x27c365 thru 0x27d703 (0x139f bytes) 000375[0] 1789124612.834778 read 0x4f5195 thru 0x4f7124 (0x1f90 bytes) 000376[1] 1789124612.848932 read 0x39d596 thru 0x39eb27 (0x1592 bytes) 000378[0] 1789124613.047553 trunc from 0x74f63a to 0x51c851 (0x232de8 bytes) 000379[0] 1789124613.197329 mapwrite 0x66dbe0 thru 0x66dc4c (0x6d bytes) 000380[1] 1789124613.322159 read 0x44b83f thru 0x44d6b5 (0x1e77 bytes) 000381[0] 1789124613.395131 trunc from 0x66dc4d to 0x9c0bd7 (0x352f8b bytes) 000383[0] 1789124613.551841 mapread 0x81e4ff thru 0x81ec4b (0x74d bytes) 000384[1] 1789124613.595839 write 0x44ed4c thru 0x44fbe2 (0xe97 bytes) 000386[1] 1789124613.706982 write 0x2dfa4c thru 0x2e15db (0x1b90 bytes) 000387[1] 1789124613.771039 read 0x781779 thru 0x781fa8 (0x830 bytes) 000390[0] 1789124613.826034 read 0x648db9 thru 0x649ef8 (0x1140 bytes) 000391[0] 1789124613.875125 mapwrite 0x3fd9a5 thru 0x3fefa3 (0x15ff bytes) 000392[1] 1789124614.037440 mapwrite 0x6f0371 thru 0x6f0919 (0x5a9 bytes) 000396[0] 1789124614.421157 mapwrite 0x8a9df0 thru 0x8aa5cf (0x7e0 bytes) 000397[0] 1789124614.561008 write 0x30eb17 thru 0x30ec63 (0x14d bytes) 000398[1] 1789124614.581319 write 0x8cef6d thru 0x8d0202 (0x1296 bytes) 000400[0] 1789124614.664521 trunc from 0x9c0bd8 to 0x2720bd (0x74eb1a bytes) 000401[0] 1789124614.763920 read 0x107d72 thru 0x109396 (0x1625 bytes) 000403[1] 1789124614.841127 trunc from 0x2720be to 0x03b1f8 (0x236ec5 bytes) 000405[0] 1789124614.943780 trunc from 0x03b1f9 to 0x4d4790 (0x499598 bytes) 000408[0] 1789124615.025790 read 0x32b212 thru 0x32c4e8 (0x12d7 bytes) 000411[0] 1789124615.069567 read 0x4a74d1 thru 0x4a87fc (0x132c bytes) 000415[0] 1789124615.141179 mapread 0x426516 thru 0x428080 (0x1b6b bytes) 000416[1] 1789124615.194558 write 0x196c12 thru 0x197663 (0xa52 bytes) 000420[1] 1789124615.337772 mapwrite 0x95c2e7 thru 0x95d1e4 (0xefe bytes) 000421[1] 1789124615.500584 trunc from 0x95d1e5 to 0x3a23ed (0x5badf7 bytes) 000422[0] 1789124615.588700 trunc from 0x3a23ee to 0x36520f (0x3d1de bytes) 000424[1] 1789124615.762713 write 0x241977 thru 0x242608 (0xc92 bytes) 000428[1] 1789124615.957977 read 0x08ceb8 thru 0x08d0ed (0x236 bytes) 000429[0] 1789124616.094529 write 0x033dde thru 0x0342ed (0x510 bytes) 000430[0] 1789124616.123021 write 0x6cb8de thru 0x6cbdc1 (0x4e4 bytes) 000431[1] 1789124616.126574 trunc from 0x6cbdc2 to 0x1efdcd (0x4dbff4 bytes) 000437[1] 1789124616.221528 mapwrite 0x050110 thru 0x0514f2 (0x13e3 bytes) 000438[0] 1789124616.314944 trunc from 0x1efdce to 0x27560c (0x8583f bytes) 000440[1] 1789124616.505951 trunc from 0x27560d to 0x905430 (0x68fe24 bytes) 000441[0] 1789124616.626687 mapwrite 0x66b737 thru 0x66d21b (0x1ae5 bytes) 000445[1] 1789124616.915544 trunc from 0x905431 to 0x205a3b (0x6ff9f5 bytes) 000447[0] 1789124617.033389 write 0x4f6c42 thru 0x4f7b21 (0xee0 bytes) 000449[0] 1789124617.100746 write 0x2a4517 thru 0x2a5ab6 (0x15a0 bytes) 000450[0] 1789124617.143731 mapwrite 0x29fb00 thru 0x2a0996 (0xe97 bytes) 000455[1] 1789124617.206660 trunc from 0x4f7b22 to 0x95568b (0x45db6a bytes) 000456[1] 1789124617.271961 trunc from 0x95568c to 0x0e84f3 (0x86d198 bytes) 000458[1] 1789124617.299374 write 0x3e473e thru 0x3e5096 (0x959 bytes) 000459[1] 1789124617.303089 write 0x752cc1 thru 0x754873 (0x1bb3 bytes) 000466[1] 1789124617.530772 mapwrite 0x8e9127 thru 0x8e9f26 (0xe00 bytes) 000467[0] 1789124617.659104 mapread 0x727dbf thru 0x728104 (0x346 bytes) 000469[1] 1789124617.821993 trunc from 0x8e9f27 to 0x52c3e9 (0x3bdb3d bytes) 000471[1] 1789124617.958122 mapwrite 0x12e350 thru 0x12e863 (0x514 bytes) 000475[0] 1789124618.153120 mapwrite 0x916658 thru 0x917d47 (0x16f0 bytes) 000477[0] 1789124618.338939 trunc from 0x917d48 to 0x523e8b (0x3f3ebc bytes) 000478[0] 1789124618.359730 read 0x4e324b thru 0x4e4324 (0x10da bytes) 000479[0] 1789124618.433790 mapwrite 0x6137a4 thru 0x614d3f (0x159c bytes) 000482[0] 1789124618.580886 read 0x034652 thru 0x035361 (0xd10 bytes) 000483[1] 1789124618.586719 mapread 0x34c819 thru 0x34d41a (0xc02 bytes) 000485[1] 1789124618.639935 write 0x149b91 thru 0x149ea7 (0x317 bytes) 000488[0] 1789124618.718488 write 0x2f2e6b thru 0x2f330c (0x4a2 bytes) 000493[0] 1789124618.811931 read 0x22326d thru 0x2237b3 (0x547 bytes) 000496[0] 1789124618.840944 mapread 0x58f022 thru 0x58f258 (0x237 bytes) 000499[1] 1789124618.873845 write 0x040a16 thru 0x041ed8 (0x14c3 bytes) All operations completed A-OK! run again with default file layout /mnt/lustre2/f16b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 20 0x14 0x280000400 Chance of close/open is 1 in 50 Seed set to 4621 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x711cdc 000002[1] 1789124622.498952 trunc from 00000000 to 0x711cdb (0x711cdc bytes) 000003[0] 1789124622.555569 mapread 0x5675ac thru 0x567f82 (0x9d7 bytes) 000004[0] 1789124622.603179 mapwrite 0x220001 thru 0x2203aa (0x3aa bytes) 000009[0] 1789124622.770919 mapwrite 0x77d05e thru 0x77ec36 (0x1bd9 bytes) truncating to largest ever: 0x96f60e 000010[1] 1789124622.909001 trunc from 0x77ec37 to 0x96f60d (0x1f09d7 bytes) 000013[1] 1789124623.055186 write 0x4389bc thru 0x438efd (0x542 bytes) 000014[0] 1789124623.099626 read 0x9261d9 thru 0x927eb5 (0x1cdd bytes) 000016[1] 1789124623.167266 write 0x859697 thru 0x85a1d5 (0xb3f bytes) 000018[1] 1789124623.210150 trunc from 0x96f60e to 0x25cab5 (0x712b58 bytes) 000020[0] 1789124623.298137 write 0x969426 thru 0x96abfa (0x17d5 bytes) 000021[0] 1789124623.332324 write 0x99499e thru 0x996531 (0x1b94 bytes) 000022[1] 1789124623.353513 read 0x63d9a5 thru 0x63ee9a (0x14f6 bytes) 000026[1] 1789124623.405094 read 0x3fe945 thru 0x400292 (0x194e bytes) 000027[1] 1789124623.416751 mapread 0x517367 thru 0x51795b (0x5f5 bytes) 000031[0] 1789124623.424609 write 0x364d8a thru 0x3659f4 (0xc6b bytes) 000032[0] 1789124623.453834 write 0x61c495 thru 0x61cdac (0x918 bytes) 000034[0] 1789124623.462275 mapwrite 0x372f50 thru 0x3731c0 (0x271 bytes) 000035[1] 1789124623.582943 mapread 0x4ccb24 thru 0x4ce460 (0x193d bytes) 000036[0] 1789124623.622914 mapwrite 0x7b73e9 thru 0x7b87da (0x13f2 bytes) 000037[0] 1789124623.732744 mapwrite 0x112aa0 thru 0x113387 (0x8e8 bytes) 000038[1] 1789124623.830559 mapread 0x458d7d thru 0x45aac3 (0x1d47 bytes) 000042[1] 1789124623.917389 write 0x7005ff thru 0x700f6f (0x971 bytes) 000045[0] 1789124623.958117 mapwrite 0x8d1225 thru 0x8d247a (0x1256 bytes) 000047[0] 1789124624.147452 mapwrite 0x5e2c07 thru 0x5e4480 (0x187a bytes) 000048[1] 1789124624.226009 mapwrite 0x8ece62 thru 0x8ecf08 (0xa7 bytes) 000049[1] 1789124624.314188 mapwrite 0x7e659b thru 0x7e66e0 (0x146 bytes) 000050[1] 1789124624.370891 read 0x940593 thru 0x942357 (0x1dc5 bytes) 000052[1] 1789124624.414884 trunc from 0x996532 to 0x55f34b (0x4371e6 bytes) 000053[1] 1789124624.458813 mapread 0x520979 thru 0x521536 (0xbbe bytes) 000056[1] 1789124624.485801 write 0x3c07c6 thru 0x3c07d7 (0x12 bytes) 000064[0] 1789124624.598160 mapread 0x086fed thru 0x087dba (0xdce bytes) 000066[0] 1789124624.658589 mapread 0x2888fe thru 0x2897bb (0xebe bytes) 000067[1] 1789124624.682594 read 0x29036d thru 0x2916f8 (0x138c bytes) 000068[1] 1789124624.698520 mapwrite 0x269dfc thru 0x26a6e8 (0x8ed bytes) 000071[0] 1789124624.876991 read 0x01647f thru 0x017b89 (0x170b bytes) 000072[0] 1789124624.904940 read 0x1d0007 thru 0x1d0c90 (0xc8a bytes) 000073[0] 1789124624.911417 write 0x7cc344 thru 0x7cd1f1 (0xeae bytes) 000075[1] 1789124624.959350 write 0x0ae297 thru 0x0af459 (0x11c3 bytes) 000076[0] 1789124625.048007 mapread 0x5d45c7 thru 0x5d5e57 (0x1891 bytes) 000077[1] 1789124625.169381 read 0x5ea377 thru 0x5eaa96 (0x720 bytes) 000078[0] 1789124625.185161 write 0x3c9395 thru 0x3ca13f (0xdab bytes) 000080[0] 1789124625.323086 read 0x70e40f thru 0x70f690 (0x1282 bytes) 000081[1] 1789124625.393597 mapwrite 0x9cbed3 thru 0x9cd309 (0x1437 bytes) 000082[0] 1789124625.679351 write 0x1cabde thru 0x1cb81c (0xc3f bytes) 000083[1] 1789124625.781891 read 0x1e84a9 thru 0x1e9a62 (0x15ba bytes) 000084[0] 1789124625.841327 write 0x3008e6 thru 0x302053 (0x176e bytes) 000087[0] 1789124625.959736 trunc from 0x9cd30a to 0x5854b8 (0x447e51 bytes) 000088[1] 1789124626.018006 mapread 0x2afde1 thru 0x2b1287 (0x14a7 bytes) 000089[1] 1789124626.114145 write 0x4da96b thru 0x4dc5f3 (0x1c89 bytes) 000090[0] 1789124626.215845 read 0x4fd2c9 thru 0x4ff20e (0x1f46 bytes) 000092[1] 1789124626.257589 mapread 0x038a53 thru 0x03973b (0xce9 bytes) 000094[1] 1789124626.269579 trunc from 0x5854b9 to 0x70c568 (0x1870b0 bytes) 000095[0] 1789124626.313570 mapwrite 0x8092eb thru 0x809b0c (0x822 bytes) 000096[0] 1789124626.406152 write 0x46645a thru 0x466b43 (0x6ea bytes) 000098[1] 1789124626.426872 write 0x40c73b thru 0x40cd46 (0x60c bytes) 000100[1] 1789124626.460032 mapread 0x252255 thru 0x25417b (0x1f27 bytes) 000103[0] 1789124626.487953 read 0x2a8de8 thru 0x2a9e17 (0x1030 bytes) 000105[0] 1789124626.513849 mapwrite 0x778a54 thru 0x77a825 (0x1dd2 bytes) 000106[0] 1789124626.592399 mapread 0x571b03 thru 0x573817 (0x1d15 bytes) 000108[0] 1789124626.624788 mapread 0x2b9609 thru 0x2bb275 (0x1c6d bytes) 000110[0] 1789124626.738298 mapread 0x47074b thru 0x471444 (0xcfa bytes) 000112[0] 1789124626.789463 trunc from 0x809b0d to 0x06d6e4 (0x79c428 bytes) 000113[0] 1789124626.848649 trunc from 0x06d6e5 to 0x01bbcb (0x51b19 bytes) 000115[1] 1789124626.923362 mapwrite 0x108a40 thru 0x108f29 (0x4ea bytes) 000116[1] 1789124627.042771 mapwrite 0x5878eb thru 0x587c4b (0x361 bytes) 000117[1] 1789124627.172672 write 0x513888 thru 0x514f09 (0x1682 bytes) 000118[0] 1789124627.230735 write 0x60026e thru 0x6014be (0x1251 bytes) 000122[1] 1789124627.300784 write 0x358c32 thru 0x35917e (0x54d bytes) 000128[1] 1789124627.567019 write 0x399f5a thru 0x39bbc2 (0x1c69 bytes) 000129[0] 1789124627.616104 write 0x0b0c2e thru 0x0b18c0 (0xc93 bytes) 000130[0] 1789124627.672025 mapwrite 0x168e7c thru 0x169c74 (0xdf9 bytes) 000132[1] 1789124627.754669 mapwrite 0x3c48bb thru 0x3c676a (0x1eb0 bytes) 000134[1] 1789124627.866571 mapread 0x0dc5da thru 0x0dd2e2 (0xd09 bytes) 000136[1] 1789124627.921940 write 0x2a8e4e thru 0x2aade3 (0x1f96 bytes) 000137[1] 1789124627.975158 mapwrite 0x2ce0af thru 0x2cf3a6 (0x12f8 bytes) 000138[0] 1789124628.119712 mapread 0x34e027 thru 0x34f521 (0x14fb bytes) 000139[0] 1789124628.186934 mapread 0x177c55 thru 0x1786c3 (0xa6f bytes) 000140[0] 1789124628.196512 mapread 0x0e45f2 thru 0x0e4cf0 (0x6ff bytes) 000141[0] 1789124628.214519 mapwrite 0x7e3b9a thru 0x7e55dc (0x1a43 bytes) 000142[0] 1789124628.530818 write 0x59dd7b thru 0x59fc1c (0x1ea2 bytes) 000147[0] 1789124628.573035 write 0x4bc166 thru 0x4bc875 (0x710 bytes) truncating to largest ever: 0x9821a8 000148[1] 1789124628.590147 trunc from 0x7e55dd to 0x9821a7 (0x19cbcb bytes) 000152[1] 1789124628.715323 mapwrite 0x1ee372 thru 0x1ef2e8 (0xf77 bytes) 000153[0] 1789124628.784436 read 0x16cbda thru 0x16cf4e (0x375 bytes) 000154[0] 1789124628.801087 write 0x42091b thru 0x422177 (0x185d bytes) 000157[0] 1789124628.867882 trunc from 0x9821a8 to 0x145c81 (0x83c526 bytes) 000159[1] 1789124628.913131 read 0x0b263f thru 0x0b3555 (0xf17 bytes) 000161[1] 1789124629.015945 mapwrite 0x79de63 thru 0x79e45f (0x5fd bytes) 000164[1] 1789124629.366252 trunc from 0x79e460 to 0x0d52f3 (0x6c916c bytes) 000165[1] 1789124629.417195 mapwrite 0x42e6f9 thru 0x43057a (0x1e82 bytes) 000166[1] 1789124629.518589 write 0x779fda thru 0x77bded (0x1e14 bytes) 000167[0] 1789124629.526691 mapwrite 0x51ec57 thru 0x520445 (0x17ef bytes) 000168[1] 1789124629.662859 read 0x70bace thru 0x70d43c (0x196f bytes) 000169[1] 1789124629.686776 read 0x21bc45 thru 0x21bff3 (0x3af bytes) 000171[1] 1789124629.698600 read 0x62430b thru 0x625353 (0x1049 bytes) 000172[1] 1789124629.705780 write 0x2ce4f1 thru 0x2d046c (0x1f7c bytes) 000173[0] 1789124629.746024 mapread 0x42b621 thru 0x42bd72 (0x752 bytes) 000175[1] 1789124629.785256 write 0x1d5828 thru 0x1d5dad (0x586 bytes) 000176[1] 1789124629.826111 trunc from 0x77bdee to 0x3964b4 (0x3e5939 bytes) 000177[1] 1789124629.865003 write 0x37353a thru 0x374566 (0x102d bytes) 000178[0] 1789124629.880773 read 0x07dad1 thru 0x07e413 (0x943 bytes) 000180[1] 1789124629.976165 write 0x74737d thru 0x747f3f (0xbc3 bytes) 000181[0] 1789124629.996643 mapread 0x21d598 thru 0x21ed22 (0x178b bytes) 000183[1] 1789124630.064704 mapread 0x46d0bd thru 0x46d152 (0x96 bytes) 000184[1] 1789124630.080691 trunc from 0x747f40 to 0x8edbd9 (0x1a5c9a bytes) 000185[1] 1789124630.137716 write 0x272549 thru 0x273be0 (0x1698 bytes) 000186[0] 1789124630.169439 read 0x86d915 thru 0x86dd48 (0x434 bytes) 000188[0] 1789124630.207441 write 0x4215aa thru 0x421983 (0x3da bytes) 000197[0] 1789124630.372613 write 0x65400f thru 0x655a9b (0x1a8d bytes) truncating to largest ever: 0x992bc4 000199[1] 1789124630.462099 trunc from 0x8edbda to 0x992bc3 (0xa4fea bytes) 000200[1] 1789124630.531925 trunc from 0x992bc4 to 0x55b697 (0x43752c bytes) 000203[0] 1789124630.614624 read 0x1e2e57 thru 0x1e4c59 (0x1e03 bytes) 000204[0] 1789124630.665540 read 0x335efb thru 0x3377e8 (0x18ee bytes) 000206[1] 1789124630.682779 read 0x320723 thru 0x3219d0 (0x12ae bytes) 000207[0] 1789124630.694113 mapread 0x36fa9a thru 0x36fc72 (0x1d9 bytes) 000209[1] 1789124630.710264 write 0x6cc8c4 thru 0x6ce354 (0x1a91 bytes) 000211[0] 1789124630.741122 mapwrite 0x0f6262 thru 0x0f7027 (0xdc6 bytes) 000212[1] 1789124630.913743 mapwrite 0x67235b thru 0x672fae (0xc54 bytes) 000213[0] 1789124630.994987 read 0x305331 thru 0x3058f0 (0x5c0 bytes) 000214[0] 1789124631.019437 read 0x0ec285 thru 0x0ec5e9 (0x365 bytes) 000215[1] 1789124631.025646 mapread 0x17674e thru 0x177c63 (0x1516 bytes) 000217[0] 1789124631.039666 mapread 0x4f0e7c thru 0x4f2d62 (0x1ee7 bytes) 000218[0] 1789124631.058877 trunc from 0x6ce355 to 0x35f91a (0x36ea3a bytes) 000219[1] 1789124631.120810 trunc from 0x35f91b to 0x049ff9 (0x315921 bytes) 000220[1] 1789124631.174023 read 0x017c68 thru 0x017d4a (0xe3 bytes) 000221[1] 1789124631.194631 trunc from 0x049ffa to 0x2a8048 (0x25e04f bytes) 000223[0] 1789124631.255199 mapread 0x247174 thru 0x2485f6 (0x1483 bytes) 000224[1] 1789124631.300508 trunc from 0x2a8049 to 0x3f6b7e (0x14eb36 bytes) 000225[1] 1789124631.342202 read 0x1b0da3 thru 0x1b257e (0x17dc bytes) 000230[1] 1789124631.555148 write 0x43754d thru 0x4394ef (0x1fa3 bytes) 000231[1] 1789124631.557175 mapwrite 0x99d469 thru 0x99ef87 (0x1b1f bytes) 000236[1] 1789124631.927778 mapwrite 0x41631b thru 0x417f9c (0x1c82 bytes) 000237[0] 1789124632.070731 mapwrite 0x0d30bb thru 0x0d4091 (0xfd7 bytes) 000238[1] 1789124632.206374 read 0x779602 thru 0x77b20a (0x1c09 bytes) 000239[0] 1789124632.220615 write 0x41b0bc thru 0x41b412 (0x357 bytes) 000241[1] 1789124632.310655 mapwrite 0x3d60a7 thru 0x3d7d34 (0x1c8e bytes) 000242[1] 1789124632.393208 mapwrite 0x97923c thru 0x97a2b0 (0x1075 bytes) 000244[0] 1789124632.449308 trunc from 0x99ef88 to 0x2c1717 (0x6dd870 bytes) 000245[1] 1789124632.515039 write 0x28eb78 thru 0x28f219 (0x6a2 bytes) 000246[1] 1789124632.571093 write 0x1d2ceb thru 0x1d4446 (0x175c bytes) 000247[0] 1789124632.601131 trunc from 0x2c1718 to 0x7fa53f (0x538e28 bytes) 000248[1] 1789124632.711559 trunc from 0x7fa540 to 0x1ac663 (0x64dedc bytes) 000250[0] 1789124632.806001 read 0x025033 thru 0x0265c3 (0x1591 bytes) 000251[1] 1789124632.868966 read 0x100063 thru 0x101e4b (0x1de9 bytes) 000252[1] 1789124632.956644 trunc from 0x1ac664 to 0x0876b4 (0x124faf bytes) 000253[1] 1789124633.036052 write 0x5a9124 thru 0x5aa93c (0x1819 bytes) 000255[0] 1789124633.073620 read 0x311939 thru 0x313653 (0x1d1b bytes) 000259[1] 1789124633.097565 write 0x5b51d4 thru 0x5b6c26 (0x1a53 bytes) 000260[1] 1789124633.110294 mapread 0x0df5e0 thru 0x0e14aa (0x1ecb bytes) 000263[0] 1789124633.263881 mapwrite 0x8e8506 thru 0x8e8c21 (0x71c bytes) 000264[0] 1789124633.435657 mapwrite 0x897329 thru 0x898054 (0xd2c bytes) 000266[1] 1789124633.511168 mapwrite 0x28ae75 thru 0x28b532 (0x6be bytes) 000267[0] 1789124633.582539 write 0x1ede91 thru 0x1ee743 (0x8b3 bytes) 000270[1] 1789124633.645910 mapread 0x406a52 thru 0x407429 (0x9d8 bytes) 000271[1] 1789124633.807820 write 0x96e301 thru 0x97020d (0x1f0d bytes) 000272[1] 1789124633.859656 read 0x1607c3 thru 0x161e1a (0x1658 bytes) 000273[0] 1789124633.866002 mapwrite 0x1043a6 thru 0x105ca1 (0x18fc bytes) 000275[1] 1789124634.033434 mapwrite 0x2f4a46 thru 0x2f5959 (0xf14 bytes) 000276[0] 1789124634.194045 trunc from 0x97020e to 0x779fad (0x1f6260 bytes) 000277[1] 1789124634.238369 mapread 0x5543e1 thru 0x5547ea (0x40a bytes) 000278[0] 1789124634.268081 mapwrite 0x228026 thru 0x22979a (0x1775 bytes) 000279[0] 1789124634.370883 mapread 0x4294f4 thru 0x42b2b7 (0x1dc4 bytes) 000280[1] 1789124634.470978 mapread 0x4f274a thru 0x4f34c4 (0xd7b bytes) 000282[0] 1789124634.554154 mapwrite 0x30d6fd thru 0x30f09c (0x19a0 bytes) 000285[1] 1789124634.705073 trunc from 0x779fae to 0x58862d (0x1f1980 bytes) 000288[0] 1789124634.803397 read 0x2f9f00 thru 0x2fb589 (0x168a bytes) 000289[1] 1789124634.820204 read 0x4a9e38 thru 0x4aaaca (0xc93 bytes) 000294[0] 1789124634.830677 read 0x4ea287 thru 0x4eadd5 (0xb4f bytes) 000295[0] 1789124634.836105 read 0x5738a3 thru 0x573e4c (0x5aa bytes) 000298[0] 1789124634.846145 mapwrite 0x578066 thru 0x579252 (0x11ed bytes) 000299[1] 1789124635.025425 write 0x672cf6 thru 0x6742e4 (0x15ef bytes) 000300[0] 1789124635.047237 read 0x0181f3 thru 0x018926 (0x734 bytes) 000301[0] 1789124635.094086 read 0x315c2f thru 0x317c05 (0x1fd7 bytes) 000306[1] 1789124635.105347 trunc from 0x6742e5 to 0x00c8ca (0x667a1a bytes) 000307[1] 1789124635.150505 mapread 0x00b2d0 thru 0x00c8ca (0x15fb bytes) 000309[1] 1789124635.185178 mapread 0x002b9c thru 0x0031a9 (0x60e bytes) 000311[0] 1789124635.196434 trunc from 0x00c8cb to 0x4db3ff (0x4ceb35 bytes) 000312[0] 1789124635.242592 read 0x2e77f9 thru 0x2e80f3 (0x8fb bytes) 000313[1] 1789124635.261445 write 0x26f1e2 thru 0x270d7f (0x1b9e bytes) 000318[0] 1789124635.387003 mapread 0x420fae thru 0x4221bd (0x1210 bytes) 000321[1] 1789124635.441561 mapread 0x420826 thru 0x42190a (0x10e5 bytes) 000322[1] 1789124635.450599 trunc from 0x4db400 to 0x52e38a (0x52f8b bytes) 000324[0] 1789124635.504747 mapread 0x3fdb7e thru 0x3ff6b9 (0x1b3c bytes) 000326[0] 1789124635.533873 mapread 0x1b9361 thru 0x1b9b1d (0x7bd bytes) 000327[1] 1789124635.538608 mapwrite 0x088b84 thru 0x08a375 (0x17f2 bytes) 000328[0] 1789124635.625763 mapwrite 0x372629 thru 0x3735aa (0xf82 bytes) 000330[0] 1789124635.732773 read 0x3de5a0 thru 0x3df57f (0xfe0 bytes) 000331[1] 1789124635.764022 mapwrite 0x0e6ae7 thru 0x0e7493 (0x9ad bytes) 000333[0] 1789124635.891313 trunc from 0x52e38b to 0x6e42ed (0x1b5f63 bytes) 000334[0] 1789124635.922451 read 0x5cf0d0 thru 0x5cf4d7 (0x408 bytes) 000335[1] 1789124635.926478 mapread 0x5271a2 thru 0x5284d8 (0x1337 bytes) 000337[1] 1789124635.984013 write 0x4ca955 thru 0x4cabf9 (0x2a5 bytes) 000338[1] 1789124636.007336 write 0x6e429a thru 0x6e484d (0x5b4 bytes) 000340[1] 1789124636.015574 mapwrite 0x160113 thru 0x1615d5 (0x14c3 bytes) 000342[1] 1789124636.076770 mapwrite 0x0b50bb thru 0x0b5cb4 (0xbfa bytes) 000345[0] 1789124636.159097 mapwrite 0x143b53 thru 0x14409e (0x54c bytes) 000346[0] 1789124636.247602 mapread 0x00b985 thru 0x00c172 (0x7ee bytes) 000347[0] 1789124636.292717 mapwrite 0x39f255 thru 0x39f70c (0x4b8 bytes) 000348[0] 1789124636.395639 mapwrite 0x6ed470 thru 0x6ed968 (0x4f9 bytes) 000349[1] 1789124636.487331 read 0x1ad805 thru 0x1af0c3 (0x18bf bytes) 000351[0] 1789124636.527553 mapread 0x58218d thru 0x582f2f (0xda3 bytes) 000353[1] 1789124636.541862 trunc from 0x6ed969 to 0x5b353f (0x13a429 bytes) 000354[0] 1789124636.583038 read 0x189101 thru 0x189616 (0x516 bytes) 000356[0] 1789124636.614082 trunc from 0x5b3540 to 0x429a71 (0x189ace bytes) 000359[0] 1789124636.669142 mapwrite 0x857cf5 thru 0x858747 (0xa53 bytes) 000361[0] 1789124636.819096 read 0x664f15 thru 0x665d81 (0xe6d bytes) 000362[1] 1789124636.837675 mapwrite 0x5a61a3 thru 0x5a6ab9 (0x917 bytes) 000363[0] 1789124636.956007 trunc from 0x858748 to 0x2562bb (0x60248c bytes) 000365[0] 1789124637.032432 trunc from 0x2562bc to 0x632aab (0x3dc7f0 bytes) 000366[0] 1789124637.059127 mapwrite 0x133ea2 thru 0x134224 (0x383 bytes) 000368[0] 1789124637.165883 write 0x1293a0 thru 0x12a1a4 (0xe05 bytes) 000369[0] 1789124637.188568 mapread 0x515a4f thru 0x51646f (0xa21 bytes) 000370[0] 1789124637.201780 write 0x6c2a61 thru 0x6c49e8 (0x1f88 bytes) 000371[1] 1789124637.204976 mapread 0x1b440f thru 0x1b5d9f (0x1991 bytes) 000372[1] 1789124637.249101 mapwrite 0x6ec414 thru 0x6edfd5 (0x1bc2 bytes) 000374[1] 1789124637.383583 mapread 0x30b44c thru 0x30ba41 (0x5f6 bytes) 000375[1] 1789124637.411065 read 0x4856c8 thru 0x485de8 (0x721 bytes) 000377[1] 1789124637.418657 write 0x539616 thru 0x53a809 (0x11f4 bytes) 000379[0] 1789124637.467172 read 0x23225a thru 0x233a36 (0x17dd bytes) 000380[0] 1789124637.514497 mapread 0x662737 thru 0x663601 (0xecb bytes) 000384[1] 1789124637.529747 write 0x9b18f4 thru 0x9b2524 (0xc31 bytes) 000385[1] 1789124637.550782 read 0x2979c0 thru 0x297f06 (0x547 bytes) 000387[0] 1789124637.555960 mapread 0x08ed6c thru 0x08ed95 (0x2a bytes) 000388[1] 1789124637.613812 trunc from 0x9b2525 to 0x3d4cd0 (0x5dd854 bytes) 000392[1] 1789124637.690115 read 0x3c55cf thru 0x3c5ff7 (0xa29 bytes) 000402[1] 1789124637.840904 trunc from 0x3d4cd1 to 0x318d64 (0xbbf6c bytes) 000403[0] 1789124637.864626 mapread 0x2b0c51 thru 0x2b289e (0x1c4e bytes) 000404[1] 1789124637.916869 mapwrite 0x12d6b3 thru 0x12eac4 (0x1412 bytes) 000407[1] 1789124638.088682 read 0x2d4a02 thru 0x2d6384 (0x1983 bytes) 000409[0] 1789124638.181792 mapwrite 0x3a117e thru 0x3a2843 (0x16c6 bytes) 000410[0] 1789124638.404736 mapwrite 0x09bd87 thru 0x09cedd (0x1157 bytes) 000412[0] 1789124638.516908 mapwrite 0x4f5e69 thru 0x4f6233 (0x3cb bytes) 000415[0] 1789124638.722938 trunc from 0x4f6234 to 0x68ad24 (0x194af1 bytes) 000416[0] 1789124638.782848 read 0x3f6de6 thru 0x3f8624 (0x183f bytes) 000417[0] 1789124638.794848 write 0x766839 thru 0x766ad7 (0x29f bytes) 000419[1] 1789124638.839040 read 0x23af39 thru 0x23c456 (0x151e bytes) 000420[0] 1789124638.859752 mapread 0x04c96a thru 0x04cd1b (0x3b2 bytes) 000421[0] 1789124638.864303 trunc from 0x766ad8 to 0x652173 (0x114964 bytes) 000424[0] 1789124638.887753 trunc from 0x652174 to 0x345f28 (0x30c24b bytes) 000426[0] 1789124638.906309 mapread 0x11b8c6 thru 0x11cfac (0x16e7 bytes) 000428[1] 1789124638.938575 write 0x4d87cd thru 0x4d98bf (0x10f3 bytes) 000429[0] 1789124638.958105 read 0x296554 thru 0x297b79 (0x1626 bytes) 000431[0] 1789124639.106477 mapread 0x113318 thru 0x113e13 (0xafc bytes) 000432[1] 1789124639.112746 read 0x4a3474 thru 0x4a3537 (0xc4 bytes) 000433[1] 1789124639.118652 trunc from 0x4d98c0 to 0x0f5f33 (0x3e398c bytes) 000434[0] 1789124639.189298 write 0x6a5e90 thru 0x6a7552 (0x16c3 bytes) 000435[1] 1789124639.203982 mapread 0x0a6463 thru 0x0a7f27 (0x1ac5 bytes) 000437[0] 1789124639.259988 trunc from 0x6a7553 to 0x749208 (0xa1cb6 bytes) 000438[0] 1789124639.319774 trunc from 0x749209 to 0x3fcb6f (0x34c699 bytes) 000440[1] 1789124639.379376 mapread 0x3d3aca thru 0x3d3af0 (0x27 bytes) 000441[0] 1789124639.436111 trunc from 0x3fcb70 to 0x0a5129 (0x357a46 bytes) 000443[1] 1789124639.506073 mapwrite 0x798675 thru 0x79a171 (0x1afd bytes) 000445[1] 1789124639.745106 mapread 0x33ccd9 thru 0x33cfbf (0x2e7 bytes) 000446[0] 1789124639.763301 mapread 0x651fcc thru 0x65203e (0x73 bytes) 000451[1] 1789124639.859505 mapwrite 0x726459 thru 0x726f6d (0xb15 bytes) 000453[1] 1789124640.040245 mapread 0x1f958b thru 0x1f9c77 (0x6ed bytes) 000454[0] 1789124640.050574 trunc from 0x79a172 to 0x20d4fb (0x58cc76 bytes) 000456[0] 1789124640.182669 trunc from 0x20d4fc to 0x93c6b6 (0x72f1bb bytes) 000462[0] 1789124640.319022 write 0x399ce9 thru 0x39af64 (0x127c bytes) 000463[1] 1789124640.341726 write 0x1bf33a thru 0x1c1272 (0x1f39 bytes) 000464[0] 1789124640.378682 mapread 0x0343ef thru 0x034f9b (0xbad bytes) 000465[1] 1789124640.436148 mapread 0x312597 thru 0x312a7f (0x4e9 bytes) 000467[1] 1789124640.446115 read 0x66e159 thru 0x66eac1 (0x969 bytes) 000469[1] 1789124640.455645 mapwrite 0x2f95c2 thru 0x2fb2f4 (0x1d33 bytes) 000471[0] 1789124640.649306 mapwrite 0x038933 thru 0x038e33 (0x501 bytes) 000473[0] 1789124640.820078 trunc from 0x93c6b7 to 0x1fdb6d (0x73eb49 bytes) 000476[1] 1789124640.850768 mapwrite 0x630e79 thru 0x632cdf (0x1e67 bytes) 000479[0] 1789124641.156177 read 0x29d7ce thru 0x29f462 (0x1c95 bytes) 000483[1] 1789124641.183793 write 0x632e86 thru 0x633747 (0x8c2 bytes) 000484[0] 1789124641.279986 write 0x3922f5 thru 0x392ad1 (0x7dd bytes) 000486[0] 1789124641.408016 trunc from 0x633748 to 0x385304 (0x2ae443 bytes) 000487[1] 1789124641.447406 write 0x88a71f thru 0x88a876 (0x158 bytes) 000489[1] 1789124641.501037 trunc from 0x88a877 to 0x1b1e57 (0x6d8a1f bytes) 000490[0] 1789124641.531122 read 0x0e5297 thru 0x0e5b36 (0x8a0 bytes) 000492[0] 1789124641.558977 mapwrite 0x3f97db thru 0x3f9eb5 (0x6db bytes) 000496[0] 1789124641.804746 mapread 0x084384 thru 0x085edd (0x1b5a bytes) 000497[0] 1789124641.860799 trunc from 0x3f9eb6 to 0x6d1cea (0x2d7e35 bytes) 000498[0] 1789124641.891083 write 0x872d4e thru 0x874a18 (0x1ccb bytes) 000500[1] 1789124641.928559 trunc from 0x874a19 to 0x6070ab (0x26d96d bytes) All operations completed A-OK! PASS 16b (54s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 07:04:07 (1789124647) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 07:04:11 (1789124651) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.255794 s, 32.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.381005 s, 22.0 MB/s ldlm.namespaces.MGC192.168.203.125@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9130c65e9800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9130d8d03800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9130d8d03800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9130d8d03800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.6605 s, 8.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.45513 s, 30.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.64024 s, 28.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.09929 s, 12.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.00934 s, 11.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.40969 s, 30.8 MB/s PASS 16d (53s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 07:05:05 (1789124705) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.368543 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15312 s, 6.8 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 07:05:13 (1789124713) TIMEOUT PASS 16f (27s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 07:05:40 (1789124740) TIMEOUT PASS 16g (27s) == sanityn test 16h: mmap read after truncate file ======= 07:06:07 (1789124767) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 07:06:13 (1789124773) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 07:06:20 (1789124780) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.1924 s, 12.0 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.09276 s, 24.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53921 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.8632 s, 14.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63421 s, 10.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.0216 s, 13.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.59356 s, 10.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.3299 s, 19.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52106 s, 17.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11069 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.22299 s, 8.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.82493 s, 14.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.08756 s, 12.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43342 s, 10.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.63526 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.09064 s, 12.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.36383 s, 11.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.08438 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.20253 s, 8.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (37s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 07:06:57 (1789124817) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri Sep 11 07:07:43 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Fri Sep 11 07:07:44 EDT 2026 + fsx 3 done: rc=0 Fri Sep 11 07:07:44 EDT 2026 + fsx 4 done: rc=0 Fri Sep 11 07:07:44 EDT 2026 + fsx 5 done: rc=0 Fri Sep 11 07:07:44 EDT 2026 PASS 16k (56s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 07:07:53 (1789124873) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 07:08:04 (1789124884) mmap test1: basic mmap operation (PASS, 0.14805s) mmap test2: MAP_PRIVATE not write back (PASS, 0.076095s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.771s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.622s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15942s) mmap test6: check mmap write/read content on two nodes (PASS, 0.26412s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.6842s) mmap test11: multi-thread shared mmap (PASS, 1.5908s) mmap test10: mtime not change for readonly mmap access (PASS, 1.134s) PASS 18 (28s) == sanityn test 19: test concurrent uncached read races ========================================================================= 07:08:31 (1789124911) oleg325-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 07:08:37 (1789124917) PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 07:08:43 (1789124923) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 07:08:49 (1789124929) now is 1789124932 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 new atime is 1789124993 PASS 23 (70s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 07:10:00 (1789125000) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 33.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117200 400 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 402 117696 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117200 400 116800 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118098 402 117696 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117200 400 116800 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118098 402 117696 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 33792 3735552 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 07:10:08 (1789125008) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 07:10:15 (1789125015) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 07:10:22 (1789125022) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 07:10:25 (1789125025) utime: good mknod times 1789125025 <= 1789125026 <= 1789125026 for /mnt/lustre/f26a utime: good mknod times 1789125025 <= 1789125026 <= 1789125026 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 07:10:32 (1789125032) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 07:10:42 (1789125042) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 07:10:44.000000000 -0400 Birth: 2026-09-11 07:10:45.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502400 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-11 07:10:44.000000000 -0400 Birth: 2026-09-11 07:10:45.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 07:10:52 (1789125052) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 07:11:01 (1789125061) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 07:11:08 (1789125068) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg325-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.125@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 07:11:32 (1789125092) -rw-r--r-- 1 root root 0 Sep 11 07:11 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (12s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 07:11:44 (1789125104) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1225: 43478 Terminated tail -f $tfile PASS 31s (10s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 07:11:54 (1789125114) PASS 31t (8s) == sanityn test 32b: lockless i/o ======================== 07:12:02 (1789125122) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 07:12:07 (1789125127) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31906 s, 39.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.75394 s, 29.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.04029 s, 25.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.19702 s, 23.9 MB/s contention_seconds 4, contention_events 19 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.9499 s, 26.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.17627 s, 24.1 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.26741 s, 41.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.32975 s, 39.4 MB/s PASS 32c (27s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 07:12:36 (1789125156) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 07:12:39 (1789125159) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 07:12:42 (1789125162) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 07:12:46 (1789125166) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 07:12:49 (1789125169) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9130c65e9800.ost_server_uuid in IDLE state after 0 sec PASS 34 (138s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 07:15:07 (1789125307) fail_loc=0x80000317 Race attempt 0 Wait for 50777 50798 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1875: 50777 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.125@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre 192.168.203.125@tcp:/lustre 7.2G 41M 7.2G 1% /mnt/lustre2 - open/close 534 (time 1789125325.55 total 10.00 last 53.37) - open/close 1099 (time 1789125335.55 total 20.01 last 56.50) - open/close 1622 (time 1789125345.57 total 30.03 last 52.19) - open/close 2193 (time 1789125355.58 total 40.03 last 57.06) - open/close 2804 (time 1789125365.58 total 50.04 last 61.09) - open/close 3420 (time 1789125375.58 total 60.04 last 61.58) - open/close 3901 (time 1789125385.59 total 70.04 last 48.09) total: 4000 open/close in 71.69 seconds: 55.80 ops/second PASS 35 (80s) == sanityn test 36: handle ESTALE/open-unlink correctly == 07:16:27 (1789125387) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg325-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.731613 s, 71.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(0) *** before(7496704) after_dd(7429120) after(7480320) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.532367 s, 98.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7480320) after_dd(7436288) after(7487488) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.704757 s, 74.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7487488) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.6058 s, 86.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7489536) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.656691 s, 79.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7489536) after_dd(7431168) after(7482368) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.697955 s, 75.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(5) *** before(7482368) after_dd(7433216) after(7484416) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.503456 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7484416) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.614142 s, 85.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7489536) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.804186 s, 65.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7491584) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.495122 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7421952) after(7473152) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.750862 s, 69.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7473152) after_dd(7438336) after(7489536) PASS 36 (301s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 07:21:28 (1789125688) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 - create 1113 (time 1789125703.42 total 10.01 last 111.14) - create 2275 (time 1789125713.44 total 20.03 last 116.05) - create 3367 (time 1789125723.44 total 30.03 last 109.17) - create 4177 (time 1789125733.47 total 40.06 last 80.79) - create 5107 (time 1789125743.47 total 50.06 last 92.99) - create 6043 (time 1789125753.48 total 60.06 last 93.53) - create 7140 (time 1789125763.48 total 70.07 last 109.65) - create 8313 (time 1789125773.48 total 80.07 last 117.26) - create 9574 (time 1789125783.49 total 90.08 last 126.03) total: 10000 create in 93.62 seconds: 106.82 ops/second PASS 37 (108s) == sanityn test 39a: file mtime does not change after rename ========================================================== 07:23:16 (1789125796) repeat after cancel_lru_locks PASS 39a (9s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 07:23:25 (1789125805) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 07:23:34 (1789125814) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 07:23:43 (1789125823) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 07:23:53 (1789125833) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 07:24:12 (1789125852) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 07:24:32 (1789125872) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (22s) == sanityn test 40e: pdirops: rename and others ======================================================================== 07:24:53 (1789125893) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 07:25:11 (1789125911) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (12s) == sanityn test 41b: pdirops: create vs create ======================================================================== 07:25:24 (1789125924) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 07:25:36 (1789125936) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (11s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 07:25:47 (1789125947) Conflict PASS 41d (12s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 07:25:59 (1789125959) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 07:26:12 (1789125972) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 07:26:24 (1789125984) Conflict PASS 41g (11s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 07:26:35 (1789125995) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2447: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 07:26:47 (1789126007) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1040s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 07:44:07 (1789127047) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 07:44:20 (1789127060) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 07:44:32 (1789127072) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 07:44:45 (1789127085) Conflict PASS 42d (17s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 07:45:02 (1789127102) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 07:45:16 (1789127116) Conflict PASS 42f (12s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 07:45:28 (1789127128) Conflict PASS 42g (11s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 07:45:39 (1789127139) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 07:45:51 (1789127151) PASS 43a (65s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 07:46:56 (1789127216) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 07:47:10 (1789127230) Conflict PASS 43c (15s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 07:47:24 (1789127244) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 07:47:37 (1789127257) Conflict PASS 43e (11s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 07:47:49 (1789127269) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (12s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 07:48:01 (1789127281) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (11s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 07:48:12 (1789127292) Conflict PASS 43h (13s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 07:48:25 (1789127305) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 07:48:29 (1789127309) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (109s) == sanityn test 43k: unlink vs create ==================== 07:50:18 (1789127418) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1119s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 08:08:57 (1789128537) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Sep 11 08:09:05 EDT 2026 PASS 44a (13s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 08:09:10 (1789128550) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 08:09:24 (1789128564) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 08:09:39 (1789128579) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 08:09:53 (1789128593) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 08:10:07 (1789128607) Conflict PASS 44f (14s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 08:10:21 (1789128621) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 08:10:36 (1789128636) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 08:10:52 (1789128652) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 08:10:56 (1789128656) PASS 45a (130s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 08:13:06 (1789128786) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 08:13:20 (1789128800) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 08:13:34 (1789128814) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 08:13:48 (1789128828) Conflict PASS 45e (13s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 08:14:01 (1789128841) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (16s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 08:14:17 (1789128857) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 08:14:33 (1789128873) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 08:14:47 (1789128887) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 08:14:50 (1789128890) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1207s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 08:34:57 (1789130097) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 08:35:11 (1789130111) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 08:35:26 (1789130126) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (17s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 08:35:42 (1789130142) Conflict PASS 46d (15s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 08:35:57 (1789130157) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 08:36:09 (1789130169) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 08:36:22 (1789130182) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 08:36:36 (1789130196) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 08:36:50 (1789130210) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 08:36:54 (1789130214) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 08:36:57 (1789130217) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 08:37:01 (1789130221) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 08:37:04 (1789130224) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 08:37:07 (1789130227) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 08:37:10 (1789130230) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 08:37:13 (1789130233) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (4s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 08:37:17 (1789130237) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.199323 s, 51.4 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (12s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 08:37:29 (1789130249) 0+1 records in 0+1 records out 158 bytes copied, 0.039603 s, 4.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3710: kill: (218785) - No such process PASS 51a (10s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 08:37:40 (1789130260) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00866708 s, 118 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685301 s, 15.3 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 08:37:59 (1789130279) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0299129 s, 34.2 kB/s PASS 51c (13s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 08:38:12 (1789130292) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050531 s, 20.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 08:38:20 (1789130300) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 08:38:29 (1789130309) llite.lustre-ffff9130c65e9800.xattr_cache=1 llite.lustre-ffff9130d8d03800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:30.000000000 -0400 Modify: 2026-09-11 08:38:30.000000000 -0400 Change: 2026-09-11 08:38:30.000000000 -0400 Birth: 2026-09-11 08:38:32.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:37.000000000 -0400 Modify: 2026-09-11 08:38:38.000000000 -0400 Change: 2026-09-11 08:38:38.000000000 -0400 Birth: 2026-09-11 08:38:38.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:43.000000000 -0400 Modify: 2026-09-11 08:38:43.000000000 -0400 Change: 2026-09-11 08:38:43.000000000 -0400 Birth: 2026-09-11 08:38:44.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505022 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:43.000000000 -0400 Modify: 2026-09-11 08:38:44.000000000 -0400 Change: 2026-09-11 08:38:44.000000000 -0400 Birth: 2026-09-11 08:38:44.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505025 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:49.000000000 -0400 Modify: 2026-09-11 08:38:50.000000000 -0400 Change: 2026-09-11 08:38:50.000000000 -0400 Birth: 2026-09-11 08:38:51.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505026 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 08:38:49.000000000 -0400 Modify: 2026-09-11 08:38:50.000000000 -0400 Change: 2026-09-11 08:38:50.000000000 -0400 Birth: 2026-09-11 08:38:51.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 08:39:00 (1789130340) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 08:39:14 (1789130354) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 08:39:29 (1789130369) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (19s) == sanityn test 55d: rename file vs link ================= 08:39:48 (1789130388) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 08:40:04 (1789130404) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 08:40:08 (1789130408) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 08:40:28 (1789130428) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 08:40:50 (1789130450) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1789130452 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.570459 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (116s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 08:42:46 (1789130566) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1789130572 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.22074 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (114s) == sanityn test 60: Verify data_version behaviour ======== 08:44:40 (1789130680) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 08:44:50 (1789130690) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 08:45:00 (1789130700) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 08:45:10 (1789130710) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0093644 s, 7.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 08:45:15 (1789130715) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0213543 s, 3.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 08:45:18 (1789130718) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 08:45:21 (1789130721) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0198352 s, 3.3 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 08:45:25 (1789130725) llite.lustre-ffff9130c65e9800.xattr_cache=1 llite.lustre-ffff9130d8d03800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 08:45:32 (1789130732) llite.lustre-ffff9130c65e9800.xattr_cache=1 llite.lustre-ffff9130d8d03800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 08:45:39 (1789130739) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.016595 s, 61.7 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0353594 s, 29.0 kB/s 231588: get lock2 231589: get lock1 231588: done 231588: get lock1 231589: done 231589: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231589: cannot get lock2: Resource deadlock avoided 231589: exit rc=0 231588: done 231588: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 08:45:51 (1789130751) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0373232 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427825 s, 24.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4393: 232170 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 08:46:04 (1789130764) opening files: ........ closing files: ........ PASS 76 (62s) == sanityn test 77a: check FIFO NRS policy =============== 08:47:06 (1789130826) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.243801 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400668 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642056 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104594 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102791 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12825 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174483 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132011 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195732 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214082 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175069 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227776 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248114 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267803 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243777 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268686 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218303 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411814 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110217 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984043 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121169 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867158 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740856 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00818753 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629818 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640771 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695947 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467395 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364173 s, 28.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210558 s, 49.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273076 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039404 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679788 s, 15.4 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 08:47:15 (1789130835) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252007 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690716 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740918 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124486 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117165 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165569 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203548 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204077 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157812 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177001 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213053 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214644 s, 4.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163135 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164376 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1745 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192457 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274131 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075999 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430635 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353129 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457246 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710375 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850918 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832252 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805282 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057728 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496074 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070322 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606122 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733192 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969903 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794369 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124706 s, 8.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234432 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090115 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162758 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168618 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147459 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207852 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117922 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125744 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248466 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172747 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236714 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215981 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222279 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208974 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151927 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287633 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247075 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368624 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458217 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108794 s, 96.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695055 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412458 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211793 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556325 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512438 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809931 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124584 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0033073 s, 317 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403914 s, 260 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00789271 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00970545 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265302 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915388 s, 11.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 08:47:28 (1789130848) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.269443 s, 62.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328076 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290162 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713102 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927701 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108283 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140526 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143296 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165755 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140788 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180971 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165836 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1919 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213496 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161179 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186878 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215358 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119492 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150644 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876696 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796841 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856848 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105616 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128854 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658752 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679761 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603892 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055287 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459061 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526824 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429696 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306712 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385047 s, 27.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.276691 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192427 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695583 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528438 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844884 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109379 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185159 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197003 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112491 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178863 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198839 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236866 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140403 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23103 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167151 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146831 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279416 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966663 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157979 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00725088 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108844 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132699 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191189 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901525 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626156 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248772 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830138 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797012 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574092 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481921 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376365 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240472 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588954 s, 17.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.429089 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644683 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311819 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871385 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261095 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255408 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181558 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203104 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186933 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326074 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297342 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325952 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205311 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246639 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299293 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229773 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.36117 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318305 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101936 s, 103 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428505 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00753438 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129675 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0027315 s, 384 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201571 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115219 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11229 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162831 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781189 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195401 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206404 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706141 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685617 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542033 s, 19.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 08:47:48 (1789130868) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.239022 s, 70.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430871 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252229 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638818 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187282 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977707 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192771 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136226 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228483 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246926 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191596 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1251 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263568 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282967 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177925 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171183 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368789 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0053419 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00923083 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110099 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204317 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086628 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963161 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0062926 s, 167 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174762 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153779 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215665 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980346 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580575 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110566 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530209 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271938 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465746 s, 22.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.549148 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380727 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106957 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101658 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103638 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134012 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229099 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178957 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157003 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299143 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270933 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280452 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331443 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404042 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37347 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421663 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499088 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842356 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167549 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643173 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707112 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700816 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755193 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109628 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109237 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119923 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192623 s, 5.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134911 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12035 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122981 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126174 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141437 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171116 s, 6.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.385609 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430493 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571471 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793022 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508731 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886375 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153942 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770085 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305342 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0366184 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139104 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163712 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291288 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289473 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238854 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225396 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285589 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477948 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809786 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463363 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541468 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679097 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657449 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073254 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117152 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778285 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999237 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981289 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932901 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101147 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110361 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119849 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131552 s, 8.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (20s) == sanityn test 77e: check TBF NID nrs policy ============ 08:48:09 (1789130889) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.25@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255588 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546882 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405273 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065018 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334892 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122841 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156256 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116629 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158512 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175682 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220061 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241665 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146063 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276648 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147059 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153559 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190758 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872553 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816932 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584062 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106617 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657127 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575886 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110684 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124019 s, 8.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071403 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810386 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795561 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128101 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121551 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139265 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943458 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102488 s, 10.2 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.232116 s, 72.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828879 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981029 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436959 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600318 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511343 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756149 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159938 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195238 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174983 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213734 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212632 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244163 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260573 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282784 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294608 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30815 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152803 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103478 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165019 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106128 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116066 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118304 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114974 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113808 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132431 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108599 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887075 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176857 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156342 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150888 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170076 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231441 s, 4.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.343979 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220716 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410738 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732865 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106042 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132609 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141121 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893792 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117477 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192657 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128278 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17579 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207167 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227039 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217826 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161942 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898602 s, 11.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035412 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267039 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290408 s, 36.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500439 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840973 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599093 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857802 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526057 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582288 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121994 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682489 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669561 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767668 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078042 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823298 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157252 s, 66.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.289261 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041044 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236694 s, 44.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426509 s, 24.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620036 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10416 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11393 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121407 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183268 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198027 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219979 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232976 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169683 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149856 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1848 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199393 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271873 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991638 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110953 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826003 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861809 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824957 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099866 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105971 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100643 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108397 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877438 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108553 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100664 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110111 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109212 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109346 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13487 s, 7.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (25s) == sanityn test 77f: check TBF JobID nrs policy ========== 08:48:33 (1789130913) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.286939 s, 58.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319021 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207117 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626352 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836847 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124347 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113227 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138736 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989782 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124774 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157216 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17547 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158977 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161258 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158021 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120206 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156023 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430697 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714476 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197374 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249476 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660971 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305591 s, 34.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878818 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660145 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428894 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434027 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551154 s, 190 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544097 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886968 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622972 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638228 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792868 s, 13.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.209116 s, 80.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370726 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423943 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402896 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676783 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143099 s, 73.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658118 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457313 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387615 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738631 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154093 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174517 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153083 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175051 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182148 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158882 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228969 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391981 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365821 s, 28.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118575 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548629 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559352 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133346 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665332 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159701 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144261 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113209 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881124 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983917 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102329 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102217 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116933 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127225 s, 8.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.301594 s, 55.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075958 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766956 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171489 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111133 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104562 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189332 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132232 s, 7.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186554 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159013 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23291 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231002 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202326 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213368 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247018 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219705 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255995 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00663787 s, 158 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00989281 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888774 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014225 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619595 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00876828 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00284249 s, 369 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994349 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128826 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762469 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114011 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777213 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187732 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987042 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114707 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915138 s, 11.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260297 s, 64.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508779 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839367 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108055 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797676 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187694 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214939 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184298 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259332 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276235 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247122 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288264 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216095 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233575 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335515 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257318 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319263 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11356 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171995 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16886 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858007 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171792 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133979 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126239 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107962 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105725 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130511 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100195 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964343 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102969 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154619 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126321 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126646 s, 8.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (22s) == sanityn test 77g: Change TBF type directly ============ 08:48:55 (1789130935) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 08:49:06 (1789130946) oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg325-client: oleg325-server: ssh exited with exit code 2 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 08:49:19 (1789130959) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (23s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 08:49:42 (1789130982) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.386632 s, 43.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274594 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140289 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382331 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850047 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278236 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903552 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687886 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785589 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28915 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317731 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261559 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42961 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39785 s, 2.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445799 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319337 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472434 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628942 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452907 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498577 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503689 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538415 s, 19.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09216 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637349 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688202 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728199 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794691 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796478 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874052 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929929 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914683 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998545 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110557 s, 9.5 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.96188 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6496 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (67s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 08:50:49 (1789131049) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26796 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920636 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135971 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163397 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11859 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209332 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24906 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121884 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238736 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816864 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114112 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230981 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145669 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163468 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302231 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381145 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246377 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863115 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909139 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889296 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922732 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827561 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736462 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576004 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112798 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877841 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842306 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132459 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104509 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872521 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888619 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113215 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136287 s, 7.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0562 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.967 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29828 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475995 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990973 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961513 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107323 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109026 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589693 s, 17.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147174 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167871 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146742 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14067 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164115 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195215 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182447 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184859 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181852 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266419 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800481 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710877 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816481 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841841 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944889 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108641 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112843 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769687 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104625 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963691 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950815 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118801 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117551 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106166 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116421 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131678 s, 8.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8876 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9479 s, 5.3 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (156s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 08:53:25 (1789131205) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.26692 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964996 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724632 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113065 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11612 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396827 s, 26.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186569 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451978 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787169 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108947 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804064 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170556 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189813 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187133 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286113 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257227 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264916 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550881 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753376 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449803 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674047 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843287 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862285 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854121 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076333 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716642 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748073 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835724 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703847 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808086 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914983 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818601 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831819 s, 12.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9775 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9706 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.32308 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622581 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459821 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613778 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073152 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094215 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596359 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109103 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125093 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142338 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126569 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149919 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164088 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116883 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151596 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162634 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201659 s, 5.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739373 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807938 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778122 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081382 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699034 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651646 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101811 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799304 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964157 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874398 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11639 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981656 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101509 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107894 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113927 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119848 s, 8.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9503 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9544 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (154s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 08:55:59 (1789131359) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.436158 s, 38.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174354 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261281 s, 40.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307015 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503298 s, 20.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295647 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481981 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390353 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386894 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535981 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977653 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833558 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664672 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142391 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13593 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101558 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196351 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477183 s, 22.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719651 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463274 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518498 s, 20.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559737 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545849 s, 19.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622276 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674549 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891464 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989017 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817094 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825214 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128895 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931193 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909485 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120874 s, 8.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.47408 s, 14.0 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90306 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.25@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.25@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.384652 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280961 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485965 s, 21.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054831 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397418 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116175 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108272 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1125 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151727 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136721 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150035 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145804 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153835 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146887 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116643 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190523 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191454 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597127 s, 17.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602819 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231267 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603959 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781882 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686911 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136858 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968031 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626427 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238556 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00275563 s, 381 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816975 s, 128 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638934 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00447572 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0065688 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755425 s, 139 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80283 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95658 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.25@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.429558 s, 39.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470908 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320198 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807982 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957088 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059154 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566399 s, 18.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126217 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627456 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090348 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126502 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173519 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161394 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168443 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144588 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168869 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120128 s, 8.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656079 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688493 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692816 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801656 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472484 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900464 s, 11.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083367 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589063 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712201 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508363 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687443 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893079 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688111 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847749 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698326 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766438 s, 13.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.56165 s, 13.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13313 s, 20.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.53668 s, 31.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483846 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402901 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337542 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871646 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1113 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723564 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982122 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074867 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162062 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902543 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175628 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161342 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212319 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171933 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153224 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188036 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049425 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108794 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104466 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106871 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714521 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928435 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078903 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862712 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111678 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143416 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087057 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850303 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118489 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943143 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904051 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117428 s, 8.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.397 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92112 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.3152 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92131 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.928927 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756694 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677964 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153287 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916116 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545067 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864552 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207471 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137535 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133327 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206283 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186518 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184236 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252823 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13696 s, 7.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133877 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189018 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302959 s, 34.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461296 s, 22.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895872 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543855 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519384 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670021 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509096 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109723 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525156 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128416 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812995 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127409 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702842 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738674 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889756 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101462 s, 10.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.822 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5917 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427348 s, 39.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424388 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219026 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085774 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908147 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153102 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180196 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220701 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156024 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231456 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237368 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214048 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243303 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280504 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285519 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274906 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363673 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116341 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069168 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574449 s, 18.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606736 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610584 s, 17.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136725 s, 7.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837329 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821834 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11908 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815879 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956649 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968621 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101019 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126141 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111639 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119637 s, 8.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.4918 s, 12.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90678 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.520113 s, 32.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492348 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498317 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128334 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146122 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128002 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167675 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181545 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176323 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168533 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205582 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175915 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180586 s, 5.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189892 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206313 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240192 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313014 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299121 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780119 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939202 s, 11.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138892 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493644 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493021 s, 213 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512428 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127896 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062599 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053799 s, 19.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104595 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120014 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825527 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684644 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370522 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00988756 s, 106 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9602 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95988 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.041 s, 9.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0083 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (424s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 09:03:04 (1789131784) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.383711 s, 43.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381838 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305811 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486259 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103295 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694278 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510638 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593286 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113159 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835231 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131016 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151709 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125745 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161185 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172555 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170436 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141995 s, 7.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114851 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399374 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404955 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495777 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495749 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111548 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135051 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609151 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120526 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622357 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799716 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135988 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708821 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736876 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801615 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889265 s, 11.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.24755 s, 14.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92129 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (47s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 09:03:52 (1789131832) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.25@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.25@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.41239 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340173 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581908 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513728 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507967 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154761 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756735 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906123 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185624 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133862 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119344 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142334 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169948 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195648 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165542 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182908 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251535 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557666 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446643 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241909 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375264 s, 27.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416674 s, 25.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418093 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542397 s, 19.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552964 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587165 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620485 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643678 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633905 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694039 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611933 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723305 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078561 s, 13.3 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.28841 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94797 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (52s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 09:04:44 (1789131884) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.25@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.455646 s, 36.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204707 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411602 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696581 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343508 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572891 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201103 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181844 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153578 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220686 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298267 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187032 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210888 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327292 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241261 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294622 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304431 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c65e9800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9130c65e9800.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715948 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131077 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733276 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472751 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101724 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143175 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790925 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10467 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107001 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109692 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103455 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978158 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128717 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120709 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137005 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120697 s, 8.7 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5813 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32584 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 09:05:32 (1789131932) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.573562 s, 29.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176764 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178645 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476743 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615525 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072986 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130172 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766571 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131651 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152718 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207006 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193964 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174828 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186216 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204273 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236031 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260364 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934874 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248489 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320515 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966275 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632066 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104828 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692991 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762485 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779768 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569294 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899446 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895447 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678163 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850688 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092911 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124009 s, 8.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5963 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87031 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.11639 s, 14.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88085 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (95s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 09:07:07 (1789132027) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.02127 s, 16.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212635 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116546 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114986 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124226 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125955 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065195 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121101 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104494 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119395 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191468 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124703 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106043 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151773 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141957 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17679 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270637 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802793 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782641 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256713 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726017 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109612 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867956 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103443 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667801 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793429 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109575 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435001 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414033 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055112 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351514 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349176 s, 30.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138628 s, 7.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.922 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.785 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (74s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 09:08:21 (1789132101) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396396 s, 42.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313337 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191078 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299287 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107765 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131145 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123099 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212579 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189321 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201768 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199793 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214012 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196258 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149499 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183092 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277795 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226373 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760552 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024536 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516258 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280103 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791251 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300741 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851403 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474188 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507758 s, 20.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443269 s, 23.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941892 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100686 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649945 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633819 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707377 s, 14.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737099 s, 14.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.35813 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1323 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg325-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.571569 s, 29.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203498 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353298 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111248 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115378 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111367 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149233 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162389 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109835 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109163 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133405 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188512 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174307 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203603 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206809 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220962 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245342 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315614 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121308 s, 86.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423897 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608379 s, 17.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430302 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00230607 s, 455 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477078 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116031 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914935 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095472 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665599 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132717 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594986 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190265 s, 55.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165891 s, 63.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166714 s, 62.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6934 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92762 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6994 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (143s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 09:10:45 (1789132245) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg325-client.virtnet /mnt/lustre (opts:) Stopping client oleg325-client.virtnet /mnt/lustre opts: Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg325-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg325-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg325-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg325-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg325-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.25,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg325-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.25,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg325-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.25,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/70060/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.25,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg325-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg325-server: oleg325-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg325-client: oleg325-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre Starting client oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre Started clients oleg325-client.virtnet: 192.168.203.125@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Starting client oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 Started clients oleg325-client.virtnet: 192.168.203.125@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9130c613a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9130d0e5c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130c613a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130d0e5c000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.664652 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107013 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820961 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118365 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180534 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24844 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245846 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193036 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242426 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307453 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243724 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243116 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344561 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346605 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275482 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348245 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369211 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477167 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465564 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444762 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460678 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489387 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47124 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.479233 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481766 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481709 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.495789 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491896 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507197 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51659 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497358 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510871 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.516092 s, 2.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.66533 s, 13.7 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17798 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.640274 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489041 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648698 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450204 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929665 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817589 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216037 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166972 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2079 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235717 s, 4.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291007 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260921 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315734 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339446 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328774 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352718 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373632 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142438 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836978 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11735 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143956 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149309 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120613 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107993 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168749 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138138 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147949 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151846 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124491 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146415 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15702 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149505 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17727 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=8 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.39578 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98446 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (215s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 09:14:19 (1789132459) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (16s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 09:14:36 (1789132476) On MGS 192.168.203.125, active = nodemap.active=1 On MGS 192.168.203.125, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.664962 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842221 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148014 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10599 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236843 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255232 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164909 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205184 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193639 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24188 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26066 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276131 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313481 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213001 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272404 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354867 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342069 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484893 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.523372 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499991 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492587 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.540191 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501786 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508724 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.520136 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.524943 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.562792 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529307 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52129 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5394 s, 1.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546619 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.535702 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.553798 s, 1.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg325-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.39164 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15277 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.125, TBF.id = On MGS 192.168.203.125, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.773404 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106535 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930229 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09737 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114618 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137837 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128074 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126597 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135459 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118616 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285259 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291696 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260674 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250382 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367698 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349229 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349699 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043985 s, 23.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800943 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887905 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682259 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464098 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042685 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595585 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669688 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259302 s, 40.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101752 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620735 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871634 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710504 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194421 s, 53.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793463 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147951 s, 70.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.17215 s, 11.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.55178 s, 29.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg325-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 On MGS 192.168.203.125, active = nodemap.active=1 On MGS 192.168.203.125, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.588688 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857257 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135072 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110812 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246167 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261508 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253722 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204764 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244983 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308577 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357668 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377106 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356662 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368287 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396673 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.435868 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428142 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129752 s, 8.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216754 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143968 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221853 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146099 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162396 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154812 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163152 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163327 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242508 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178206 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180073 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201426 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205087 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19654 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205374 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=3 removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.18132 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92906 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.125, TBF.id = On MGS 192.168.203.125, active = nodemap.active=0 PASS 77kj (190s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 09:17:45 (1789132665) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 09:17:56 (1789132676) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg325-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.42526 s, 306 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg325-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1201 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (57s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 09:18:53 (1789132733) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.567407 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620946 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152704 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162794 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176545 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829106 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850321 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158725 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146715 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235697 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261292 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292123 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280143 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320951 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195013 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292441 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398145 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105169 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916135 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107853 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930591 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744628 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887218 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882059 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108568 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094977 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11838 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891012 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923535 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936357 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0921566 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104346 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115648 s, 9.1 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.08638 s, 14.8 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.1459 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.77039 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752923 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802734 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11023 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952194 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130012 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896641 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154107 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834914 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231982 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118595 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221957 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188557 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122638 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275287 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209166 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821208 s, 12.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9130c613a800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9130c613a800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408441 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449268 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447769 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407052 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410259 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417229 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416387 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42155 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.423188 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502021 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433098 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436926 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443989 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442707 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.44772 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468033 s, 2.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg325-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg325-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.21017 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17415 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (90s) == sanityn test 77o: Changing rank should not panic ====== 09:20:24 (1789132824) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 09:20:37 (1789132837) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (110s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 09:22:27 (1789132947) mds.MDS.mdt.nrs_policies=tbf oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg325-client: oleg325-server: ssh exited with exit code 36 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg325-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg325-client: oleg325-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (38s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 09:23:05 (1789132985) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273861 kill md_thread xargs PASS 77r (60s) == sanityn test 77s: Check TBF LRU shrinker ============== 09:24:05 (1789133045) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (66s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 09:25:11 (1789133111) oleg325-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg325-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg325-client: oleg325-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 09:25:21 (1789133121) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935667 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:25:23.000000000 -0400 Modify: 2026-09-11 09:25:23.000000000 -0400 Change: 2026-09-11 09:25:23.000000000 -0400 Birth: 2026-09-11 09:25:24.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (17s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 09:25:38 (1789133138) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 09:25:41 (1789133141) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 09:25:45 (1789133145) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 09:25:49 (1789133149) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 09:25:52 (1789133152) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 09:25:55 (1789133155) - open/close 638 (time 1789133170.93 total 10.01 last 63.72) - open/close 1343 (time 1789133180.94 total 20.02 last 70.42) - open/close 2065 (time 1789133190.94 total 30.02 last 72.20) total: 2500 open/close in 36.26 seconds: 68.94 ops/second snapshot_time 1789133293.575828713 secs.nsecs start_time 1789133159.811231576 secs.nsecs elapsed_time 133.764597137 secs.nsecs open 2505 samples [usecs] 140 14361 3982911 10991101463 close 2505 samples [usecs] 169 5850 933490 683950294 mknod 2501 samples [usecs] 536 14045 3472518 8510312518 unlink 2500 samples [usecs] 674 50479 4521460 16148977430 mkdir 2 samples [usecs] 5388 7302 12690 82349748 rmdir 2 samples [usecs] 11266 15543 26809 368507605 rename 2500 samples [usecs] 519 50632 16769154 217132810310 getattr 3764 samples [usecs] 10 13535 283505 658349417 parallel_rename_file 2500 samples [usecs] 519 50632 16769154 217132810310 crossdir_rename 2500 samples [usecs] 519 50632 16769154 217132810310 readpage 5 samples [usecs] 4409 30925 73462 1659512976 PASS 81d (143s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 09:28:18 (1789133298) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7627 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 09:28:25 (1789133305) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 09:28:28 (1789133308) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 09:28:43 (1789133323) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1789133324 Finishing test test1 at 1789133329 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 09:28:55 (1789133335) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 09:28:58 (1789133338) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 09:29:03 (1789133343) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 09:29:06 (1789133346) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/d93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5831 0x16c7 0x240000400 1 5758 0x167e 0x280000400 /mnt/lustre2/d93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5759 0x167f 0x280000400 0 5832 0x16c8 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 09:29:23 (1789133363) osc.lustre-OST0000-osc-ffff9130c613a800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9130d0e5c000.idle_timeout=0 osc.lustre-OST0001-osc-ffff9130c613a800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9130d0e5c000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.24164 s, 33.9 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98291 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6414: 290912 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0627602 s, 65.3 kB/s osc.lustre-OST0000-osc-ffff9130c613a800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9130d0e5c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130c613a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9130d0e5c000.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 09:29:39 (1789133379) llite.lustre-ffff9130c613a800.fast_read=0 llite.lustre-ffff9130d0e5c000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0547139 s, 38.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9130c613a800.fast_read=1 llite.lustre-ffff9130d0e5c000.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 09:29:59 (1789133399) llite.lustre-ffff9130c613a800.fast_read=0 llite.lustre-ffff9130d0e5c000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0101772 s, 1.2 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9130c613a800.fast_read=1 llite.lustre-ffff9130d0e5c000.fast_read=1 PASS 95b (18s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 09:30:17 (1789133417) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 09:30:22 (1789133422) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00663914 s, 617 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (9s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 09:30:31 (1789133431) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23088 s, 9.1 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (10s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 09:30:40 (1789133440) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.370349 s, 5.7 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (13s) == sanityn test 100e: DoM: read on open and file size ==== 09:30:53 (1789133453) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00569249 s, 2.1 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 09:31:03 (1789133463) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00865123 s, 473 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 09:31:11 (1789133471) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00896489 s, 457 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 09:31:19 (1789133479) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00807769 s, 507 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 09:31:27 (1789133487) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 54 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf54:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 09:31:35 (1789133495) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1789133498 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1789133501 Finishing test test23 at 1789133501 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 09:31:49 (1789133509) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 09:31:52 (1789133512) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 09:32:20 (1789133540) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 09:32:24 (1789133544) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0150443 s, 68.1 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:32:26.000000000 -0400 Modify: 2026-09-11 09:32:26.000000000 -0400 Change: 2026-09-11 09:32:26.000000000 -0400 Birth: 2026-09-11 09:32:27.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938203 -rw-r--r-- 1 0 0 1789133547 1789133546 1789133546 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938203 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:32:26.000000000 -0400 Modify: 2026-09-11 09:32:26.000000000 -0400 Change: 2026-09-11 09:32:26.000000000 -0400 Birth: 2026-09-11 09:32:27.000000000 -0400 1789133546 1024 11 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 09:32:33 (1789133553) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 09:32:41 (1789133561) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244065 s, 43.0 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5766 0x1686 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 First grouplock blocks second one PASS 107a (10s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 09:32:51 (1789133571) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.286519 s, 36.6 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5841 0x16d1 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7627 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 09:33:06 (1789133586) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00379158 s, 2.6 MB/s llite.lustre-ffff9130c613a800.hybrid_io=0 llite.lustre-ffff9130d0e5c000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.91539 s, 1.7 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 09:33:19 (1789133599) 192.168.203.125@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg325-client.virtnet /mnt/lustre (opts:) 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 192.168.203.125@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.125@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg325-client.virtnet /mnt/lustre (opts:) Stopping client oleg325-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg325-server' oleg325-server: oleg325-server.virtnet: executing load_modules_local oleg325-server: Loading modules from /home/green/git/lustre-release/lustre oleg325-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg325-server: detected 4 online CPUs by sysfs oleg325-server: MODOPTS_LIBCFS= oleg325-server: Force libcfs to create 2 CPU partitions Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre Starting client: oleg325-client.virtnet: -o user_xattr,flock 192.168.203.125@tcp:/lustre /mnt/lustre2 PASS 109 (46s) == sanityn test 110: do not grant another lock on resend ========================================================== 09:34:06 (1789133646) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 09:34:08.000000000 -0400 Modify: 2026-09-11 09:34:08.000000000 -0400 Change: 2026-09-11 09:34:08.000000000 -0400 Birth: 2026-09-11 09:34:09.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg325-client.virtnet oleg325-client.virtnet PASS 110 (170s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 09:36:55 (1789133815) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 09:36:59 (1789133819) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 09:37:02 (1789133822) lustre-OST0000-osc-ffff9130c369b000 active. lustre-OST0001-osc-ffff9130c369b000 active. lustre-OST0000-osc-ffff9130c369b000 active. lustre-OST0001-osc-ffff9130c369b000 active. MGC192.168.203.125@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 09:37:09 (1789133829) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 09:37:13 (1789133833) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 09:37:17 (1789133837) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 09:37:21 (1789133841) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00908925 s, 2.3 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0905945 s, 226 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 09:37:30 (1789133850) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=10 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1208 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2544 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1877 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2109 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2063 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2239 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3045 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2584 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1955 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2372 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2025 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2861 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2845 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.309 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2465 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2919 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4183 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4464 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3801 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4235 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg325-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9130c369b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9130c369b000.ost_server_uuid in FULL state after 0 sec PASS 200 (84s) cleanup: ====================================================== == sanityn test complete, duration 9817 sec ============== 09:38:56 (1789133936) osc.lustre-OST0000-osc-ffff9130c369b000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0000-osc-ffff9130c515c000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9130c369b000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9130c515c000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789133940.878924134 secs.nsecs start_time 1789132319.972230342 secs.nsecs elapsed_time 1620.906693792 secs.nsecs uncompressed_objects 1596 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789133940.878985727 secs.nsecs start_time 1789132332.681190090 secs.nsecs elapsed_time 1608.197795637 secs.nsecs uncompressed_objects 1473 samples [reqs] === sanityn: start cleanup 09:38:59 (1789133939) === Stopping clients: oleg325-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg325-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 09:44:12 (1789134252) ===