-----============= acceptance-small: sanityn ============----- Thu Aug 27 14:58:56 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg432-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 14:59:18 (1787857158) === Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Mount client oleg432-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg432-client.virtnet: executing check_config_client /mnt/lustre oleg432-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg432-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9de003eca000.idle_timeout=debug osc.lustre-OST0000-osc-ffff9de010b9d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de003eca000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de010b9d800.idle_timeout=debug disable quota as required oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:59:54 (1787857194) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250761 s, 41.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7639] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:59:57 (1787857197) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 15:00:05 (1787857205) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (8s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 15:00:13 (1787857213) -rw-r--r-- 1 root root 0 Aug 27 15:00 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 15:00:22 (1787857222) -rw-r--r-- 1 root root 0 Aug 27 15:00 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 15:00:28 (1787857228) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 15:00:35 (1787857235) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 15:00:41 (1787857241) total 1 -rwxrwxrwx 1 root root 0 Aug 27 15:00 f2a -rwxrwxrwx 1 root root 0 Aug 27 15:00 f2b total 1 -rwxrwxrwx 1 root root 0 Aug 27 15:00 f2a -rwxrwxrwx 1 root root 0 Aug 27 15:00 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0124798 s, 41.0 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 15:00:48 (1787857248) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 15:00:51 (1787857251) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354581 s, 5.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 15:00:57 (1787857257) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 15:01:04 (1787857264) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 15:01:11 (1787857271) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 15:01:17 (1787857277) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 15:01:24 (1787857284) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 15:01:32 (1787857292) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 15:01:39 (1787857299) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 15:01:46 (1787857306) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332438 s, 31.5 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0366115 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (7s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 15:01:53 (1787857313) 1+0 records in 1+0 records out 1 byte copied, 0.0170325 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0141832 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0101815 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123326 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0176857 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0200862 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184522 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0237856 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0204283 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0160088 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0247692 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.034003 s, 0.0 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 15:02:01 (1787857321) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0239746 s, 128 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.06431 s, 63.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00026334 s, 11.7 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 15:02:07 (1787857327) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 15:02:14 (1787857334) warning: '-runtime' deprecated, use '-t runtime' instead - link 332 (time 1787857347.05 total 10.00 last 33.18) - link 682 (time 1787857357.08 total 20.04 last 34.89) - link 996 (time 1787857367.09 total 30.05 last 31.37) - link 1328 (time 1787857377.10 total 40.05 last 33.20) - link 1688 (time 1787857387.10 total 50.05 last 36.00) - link 2048 (time 1787857397.11 total 60.06 last 35.95) - link 2428 (time 1787857407.13 total 70.08 last 37.93) - link 2798 (time 1787857417.13 total 80.08 last 36.99) - link 3133 (time 1787857427.14 total 90.10 last 33.46) total: 3469 link in 100.04 seconds: 34.68 ops/second using seed 3712342982 running for 100 seconds - stat 5695 (time 1787857349 ; total 11 ; last 11) - stat 10000 (time 1787857356 ; total 18 ; last 7) - stat 16065 (time 1787857367 ; total 29 ; last 11) - stat 20000 (time 1787857374 ; total 36 ; last 7) - stat 25862 (time 1787857385 ; total 47 ; last 11) - stat 29703 (time 1787857396 ; total 58 ; last 11) - stat 30000 (time 1787857396 ; total 58 ; last 0) - stat 32828 (time 1787857407 ; total 69 ; last 11) - stat 36084 (time 1787857418 ; total 80 ; last 11) - stat 39414 (time 1787857429 ; total 91 ; last 11) - stat 40000 (time 1787857431 ; total 93 ; last 2) total: 41931 stats in 100 seconds: 419.309998 stats/second - unlinked 0 (time 1787857388 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3469) error: No such file or directory total: 3468 unlinks in 87 seconds: 39.862068 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21452) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21455) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 15:04:40 (1787857480) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 15:04:49 (1787857489) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 22639 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 15:04:57 (1787857497) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 23219 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 15:05:05 (1787857505) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23797 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 15:05:14 (1787857514) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24381 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 15:05:22 (1787857522) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24964 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 15:05:30 (1787857530) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (51s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 15:06:22 (1787857582) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.89853 s, 9.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.501969 s, 16.7 MB/s Chance of close/open is 1 in 50 Seed set to 7586 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0xcaed 000001[1] 1787857587.742510 trunc from 00000000 to 0x00caec (0xcaed bytes) 000003[1] 1787857587.808647 mapwrite 0x68ce52 thru 0x697e56 (0xb005 bytes) truncating to largest ever: 0x5fdab6 000004[1] 1787857587.915338 trunc from 0x697e57 to 0x5fdab5 (0x9a3a1 bytes) 000007[1] 1787857587.986717 read 0x4ebfb3 thru 0x4f6c02 (0xac50 bytes) 000008[1] 1787857588.034461 mapwrite 0x8ce2cc thru 0x8d16b4 (0x33e9 bytes) 000009[1] 1787857588.229918 mapread 0x515649 thru 0x5186bc (0x3074 bytes) 000011[1] 1787857588.342534 read 0x3789b9 thru 0x38801f (0xf667 bytes) 000015[1] 1787857588.403296 read 0x39b4fd thru 0x3a915b (0xdc5f bytes) 000016[1] 1787857588.437433 trunc from 0x8d16b5 to 0x2f7015 (0x5da69f bytes) 000017[0] 1787857588.488671 write 0x19b1d7 thru 0x1a0f37 (0x5d61 bytes) 000018[1] 1787857588.533061 mapread 0x1be107 thru 0x1c1e2e (0x3d28 bytes) 000020[0] 1787857588.588613 mapwrite 0x9b3fd6 thru 0x9b7f14 (0x3f3f bytes) 000025[0] 1787857588.881201 mapread 0x223896 thru 0x22811a (0x4885 bytes) 000027[0] 1787857588.933088 mapwrite 0x57f3a2 thru 0x58217e (0x2ddd bytes) 000028[0] 1787857589.082079 trunc from 0x9b7f15 to 0x1dd753 (0x7da7c1 bytes) 000030[0] 1787857589.143692 mapwrite 0x79ab77 thru 0x7a8c05 (0xe08f bytes) 000032[0] 1787857589.292069 mapread 0x50d5db thru 0x5144e8 (0x6f0e bytes) 000033[1] 1787857589.353189 write 0x824c01 thru 0x82f48d (0xa88d bytes) 000034[0] 1787857589.385130 mapwrite 0x79f05a thru 0x7a5447 (0x63ee bytes) 000037[1] 1787857589.625740 read 0x26a179 thru 0x2762bd (0xc145 bytes) 000038[1] 1787857589.652853 mapread 0x7a633f thru 0x7b2a5d (0xc71f bytes) 000040[1] 1787857589.775315 trunc from 0x82f48e to 0x307a57 (0x527a36 bytes) 000044[0] 1787857589.893879 write 0x8956d0 thru 0x899d19 (0x464a bytes) 000046[0] 1787857589.970525 mapread 0x67ba15 thru 0x68999e (0xdf8a bytes) 000047[1] 1787857590.062018 read 0x123d32 thru 0x12e461 (0xa730 bytes) 000048[0] 1787857590.101029 mapread 0x13cae8 thru 0x141f34 (0x544d bytes) 000049[1] 1787857590.127938 read 0x0dbc93 thru 0x0de62e (0x299c bytes) 000050[1] 1787857590.136515 mapread 0x83a8bf thru 0x840de9 (0x652b bytes) 000052[1] 1787857590.158309 trunc from 0x899d1a to 0x5aa5c7 (0x2ef752 bytes) 000053[0] 1787857590.197328 mapwrite 0x8bb847 thru 0x8caea2 (0xf65c bytes) 000059[1] 1787857590.423655 mapwrite 0x817081 thru 0x8259c8 (0xe948 bytes) 000061[0] 1787857590.565411 mapwrite 0x25944a thru 0x261a24 (0x85db bytes) 000065[1] 1787857590.700806 mapwrite 0x92d0b2 thru 0x93a1b4 (0xd103 bytes) truncating to largest ever: 0x8fcc11 000066[1] 1787857590.862232 trunc from 0x93a1b5 to 0x8fcc10 (0x3d5a4 bytes) 000070[0] 1787857590.989129 read 0x3fd510 thru 0x3fd5c9 (0xba bytes) 000072[0] 1787857591.018489 trunc from 0x8fcc11 to 0x8448f7 (0xb8319 bytes) 000074[1] 1787857591.059522 mapwrite 0x46d611 thru 0x46e8aa (0x129a bytes) 000075[1] 1787857591.210248 write 0x50880b thru 0x5092f6 (0xaec bytes) 000080[1] 1787857591.386094 read 0x78a12c thru 0x798857 (0xe72c bytes) 000081[1] 1787857591.410390 mapwrite 0x7d3106 thru 0x7d798f (0x488a bytes) 000082[1] 1787857591.538711 trunc from 0x8448f8 to 0x4d49ad (0x36ff4a bytes) 000085[0] 1787857591.617147 mapread 0x4982a1 thru 0x4a1d90 (0x9af0 bytes) 000090[0] 1787857591.699273 mapread 0x1ee6bb thru 0x1f2971 (0x42b7 bytes) 000092[0] 1787857591.747696 read 0x070aa3 thru 0x07ea58 (0xdfb6 bytes) 000096[0] 1787857591.771123 mapread 0x12907d thru 0x135558 (0xc4dc bytes) 000098[0] 1787857591.801234 mapwrite 0x7e0954 thru 0x7e496b (0x4018 bytes) 000099[1] 1787857592.089416 trunc from 0x7e496c to 0x35af8f (0x4899dc bytes) 000100[1] 1787857592.147490 trunc from 0x35af90 to 0x64d904 (0x2f2975 bytes) 000101[0] 1787857592.202506 trunc from 0x64d905 to 0x5cb821 (0x820e3 bytes) 000106[1] 1787857592.397202 mapread 0x5c90d2 thru 0x5cb821 (0x2750 bytes) 000107[1] 1787857592.542380 trunc from 0x5cb822 to 0x8e6eeb (0x31b6ca bytes) 000108[0] 1787857592.606457 mapwrite 0x50ae67 thru 0x518f62 (0xe0fc bytes) 000109[1] 1787857592.763529 write 0x9a56ff thru 0x9b3a3c (0xe33e bytes) 000111[0] 1787857592.805664 mapread 0x65c014 thru 0x65dcb1 (0x1c9e bytes) 000113[0] 1787857592.868345 mapwrite 0x621f99 thru 0x6233d3 (0x143b bytes) 000115[1] 1787857592.979383 write 0x8e4ba1 thru 0x8e67c5 (0x1c25 bytes) truncating to largest ever: 0x952795 000117[1] 1787857593.014623 trunc from 0x9b3a3d to 0x952794 (0x612a8 bytes) 000118[1] 1787857593.068636 read 0x482ba7 thru 0x48e879 (0xbcd3 bytes) 000121[1] 1787857593.082634 write 0x441342 thru 0x44cd5b (0xba1a bytes) 000122[0] 1787857593.107703 mapwrite 0x0c52fb thru 0x0d0a65 (0xb76b bytes) 000124[0] 1787857593.257122 mapread 0x57cdee thru 0x584e16 (0x8029 bytes) 000125[1] 1787857593.338026 mapwrite 0x672505 thru 0x680efc (0xe9f8 bytes) 000126[1] 1787857593.498739 mapwrite 0x378102 thru 0x38354a (0xb449 bytes) 000127[1] 1787857593.647061 trunc from 0x952795 to 0x88d82a (0xc4f6a bytes) 000128[1] 1787857593.665852 mapread 0x7d6964 thru 0x7db105 (0x47a2 bytes) 000129[0] 1787857593.711856 mapread 0x6a1e88 thru 0x6af62a (0xd7a3 bytes) 000130[0] 1787857593.775343 mapwrite 0x366692 thru 0x36ac0d (0x457c bytes) 000135[0] 1787857594.021323 read 0x3f796f thru 0x3f9f7a (0x260c bytes) 000138[0] 1787857594.081349 trunc from 0x88d82b to 0x460a09 (0x42ce21 bytes) 000142[0] 1787857594.186217 trunc from 0x460a0a to 0x18f4c5 (0x2d1544 bytes) 000143[1] 1787857594.217091 trunc from 0x18f4c6 to 0x267705 (0xd8240 bytes) 000145[1] 1787857594.310698 write 0x79cc06 thru 0x7a4984 (0x7d7f bytes) 000146[1] 1787857594.342177 mapread 0x0b3954 thru 0x0c083e (0xceeb bytes) 000147[0] 1787857594.409472 read 0x3a9617 thru 0x3a988f (0x279 bytes) 000149[0] 1787857594.500538 mapread 0x6153fe thru 0x61b4da (0x60dd bytes) 000150[1] 1787857594.586838 mapread 0x2890b8 thru 0x28ed87 (0x5cd0 bytes) 000151[1] 1787857594.634229 mapwrite 0x9cc6a5 thru 0x9d692a (0xa286 bytes) 000153[0] 1787857594.796906 write 0x89d634 thru 0x8ac369 (0xed36 bytes) 000154[0] 1787857594.841210 write 0x0ada05 thru 0x0bca93 (0xf08f bytes) 000156[0] 1787857594.880030 write 0x3711fe thru 0x37b8c0 (0xa6c3 bytes) 000157[1] 1787857594.905750 write 0x34efd4 thru 0x34f0e0 (0x10d bytes) 000158[1] 1787857594.970325 mapread 0x5ab8db thru 0x5b4bd0 (0x92f6 bytes) 000161[1] 1787857595.061088 trunc from 0x9d692b to 0x275d24 (0x760c06 bytes) 000163[0] 1787857595.109080 trunc from 0x275d25 to 0x6efe1a (0x47a0f6 bytes) 000164[1] 1787857595.183118 mapwrite 0x36a006 thru 0x36e1ff (0x41fa bytes) 000165[0] 1787857595.423081 write 0x880d26 thru 0x88d361 (0xc63c bytes) 000167[1] 1787857595.479299 mapread 0x7a1dfa thru 0x7b13cc (0xf5d3 bytes) 000171[0] 1787857595.805710 read 0x58be2d thru 0x59041a (0x45ee bytes) 000173[0] 1787857595.829519 mapwrite 0x9dc63c thru 0x9e41f1 (0x7bb6 bytes) 000174[1] 1787857595.919916 mapread 0x990ea8 thru 0x99f0a0 (0xe1f9 bytes) 000177[1] 1787857596.023805 read 0x2e0efa thru 0x2e30ee (0x21f5 bytes) 000178[1] 1787857596.052326 mapwrite 0x3e2c1b thru 0x3e9bb1 (0x6f97 bytes) 000180[1] 1787857596.177861 read 0x2bc36e thru 0x2c98b7 (0xd54a bytes) 000182[0] 1787857596.222345 write 0x9ae244 thru 0x9b8171 (0x9f2e bytes) 000183[1] 1787857596.247964 read 0x8c3c77 thru 0x8ce27f (0xa609 bytes) 000186[1] 1787857596.295183 mapread 0x47e649 thru 0x4814e7 (0x2e9f bytes) 000187[0] 1787857596.338290 mapwrite 0x44b199 thru 0x456032 (0xae9a bytes) 000188[0] 1787857596.442406 mapread 0x22cc48 thru 0x232527 (0x58e0 bytes) 000189[0] 1787857596.463764 mapwrite 0x9f396b thru 0x9f996b (0x6001 bytes) 000190[0] 1787857596.615066 trunc from 0x9f996c to 0x5b4358 (0x445613 bytes) 000191[1] 1787857596.636779 mapwrite 0x8830ac thru 0x885f8d (0x2ee2 bytes) 000193[1] 1787857596.798240 trunc from 0x885f8e to 0x16781a (0x71e773 bytes) 000194[1] 1787857596.852737 mapwrite 0x786871 thru 0x795474 (0xec04 bytes) 000195[0] 1787857596.997884 read 0x583a73 thru 0x58e465 (0xa9f3 bytes) 000196[0] 1787857597.069573 mapwrite 0x8f6912 thru 0x8f6b2a (0x219 bytes) 000197[0] 1787857597.190994 trunc from 0x8f6b2b to 0x0c1041 (0x835ae9 bytes) 000199[1] 1787857597.232797 trunc from 0x0c1042 to 0x840eee (0x77fead bytes) 000200[0] 1787857597.279352 mapread 0x7a2a43 thru 0x7a759f (0x4b5d bytes) 000201[0] 1787857597.348258 mapwrite 0x20e639 thru 0x21d2fd (0xecc5 bytes) 000202[0] 1787857597.462488 write 0x09918d thru 0x09d695 (0x4509 bytes) 000203[0] 1787857597.510058 read 0x6062e8 thru 0x60fee2 (0x9bfb bytes) 000205[1] 1787857597.522416 trunc from 0x840eef to 0x53b48d (0x305a61 bytes) 000206[0] 1787857597.658731 write 0x889d65 thru 0x897d63 (0xdfff bytes) 000207[0] 1787857597.706378 trunc from 0x897d64 to 0x76cb81 (0x12b1e2 bytes) 000208[0] 1787857597.751671 mapread 0x0c4184 thru 0x0cde73 (0x9cf0 bytes) 000209[1] 1787857597.780089 read 0x1f02fa thru 0x1f09ca (0x6d1 bytes) 000210[1] 1787857597.830536 trunc from 0x76cb82 to 0x30334e (0x469833 bytes) 000213[0] 1787857597.923346 trunc from 0x30334f to 0x255a62 (0xad8ec bytes) 000216[0] 1787857598.104789 mapwrite 0x676500 thru 0x67acb7 (0x47b8 bytes) 000217[1] 1787857598.341995 trunc from 0x67acb8 to 0x07586e (0x605449 bytes) 000218[1] 1787857598.464529 read 0x06f3c9 thru 0x07586e (0x64a6 bytes) 000221[0] 1787857598.495417 mapwrite 0x5530aa thru 0x556b0b (0x3a62 bytes) 000222[0] 1787857598.656294 mapread 0x3a3ef0 thru 0x3a587f (0x1990 bytes) 000223[0] 1787857598.697125 mapwrite 0x4b1751 thru 0x4b99ae (0x825e bytes) 000224[0] 1787857598.766575 mapwrite 0x2e5640 thru 0x2e8d53 (0x3714 bytes) 000227[0] 1787857598.863534 read 0x39950e thru 0x39a6b5 (0x11a8 bytes) 000228[1] 1787857598.904007 mapwrite 0x5a152c thru 0x5af5ef (0xe0c4 bytes) 000229[1] 1787857599.174625 mapwrite 0x9c5c32 thru 0x9cbe01 (0x61d0 bytes) 000232[0] 1787857599.442241 read 0x9bef9c thru 0x9c7726 (0x878b bytes) 000233[1] 1787857599.519056 write 0x021ad0 thru 0x02d120 (0xb651 bytes) 000234[0] 1787857599.541493 mapread 0x2f2e59 thru 0x300bc9 (0xdd71 bytes) 000235[0] 1787857599.640379 mapwrite 0x74b3a7 thru 0x7529e8 (0x7642 bytes) truncating to largest ever: 0x9e80b2 000240[0] 1787857599.885333 trunc from 0x9cbe02 to 0x9e80b1 (0x1c2b0 bytes) 000241[0] 1787857599.961873 mapread 0x2a719b thru 0x2b0dca (0x9c30 bytes) 000245[0] 1787857600.043097 write 0x9b8e09 thru 0x9c83a5 (0xf59d bytes) 000246[1] 1787857600.057491 trunc from 0x9e80b2 to 0x7ecd43 (0x1fb36e bytes) 000247[1] 1787857600.185597 read 0x7e286f thru 0x7e463b (0x1dcd bytes) 000249[0] 1787857600.260493 mapread 0x6b8822 thru 0x6c413c (0xb91b bytes) 000250[1] 1787857600.342761 trunc from 0x7ecd44 to 0x7aa66f (0x426d4 bytes) 000251[1] 1787857600.382443 write 0x09ddd1 thru 0x09f085 (0x12b5 bytes) 000258[0] 1787857600.515691 trunc from 0x7aa670 to 0x985f1d (0x1db8ae bytes) 000259[1] 1787857600.622148 write 0x12e127 thru 0x138bc1 (0xaa9b bytes) 000260[1] 1787857600.697848 write 0x9f42b1 thru 0x9fffff (0xbd4f bytes) 000262[0] 1787857600.755103 trunc from 0xa00000 to 0x0ee537 (0x911ac8 bytes) 000264[1] 1787857600.818424 mapread 0x0877b9 thru 0x089823 (0x206b bytes) 000266[1] 1787857600.935079 mapread 0x0e5eeb thru 0x0ee537 (0x864d bytes) 000270[1] 1787857600.947658 write 0x28c527 thru 0x29b019 (0xeaf3 bytes) 000271[1] 1787857601.008099 mapwrite 0x809946 thru 0x80fbb3 (0x626e bytes) 000272[1] 1787857601.236337 mapread 0x5280da thru 0x532c23 (0xab4a bytes) 000273[0] 1787857601.264100 mapread 0x758ced thru 0x75a4ac (0x17c0 bytes) 000274[1] 1787857601.307151 trunc from 0x80fbb4 to 0x81fc30 (0x1007d bytes) 000277[1] 1787857601.394318 read 0x35ddaf thru 0x3674d0 (0x9722 bytes) 000278[0] 1787857601.430371 trunc from 0x81fc31 to 0x401052 (0x41ebde bytes) 000281[1] 1787857601.528673 mapwrite 0x4b993f thru 0x4bfdff (0x64c1 bytes) 000282[0] 1787857601.645444 read 0x480cc9 thru 0x486527 (0x585f bytes) 000283[0] 1787857601.677758 mapwrite 0x57723d thru 0x57c154 (0x4f18 bytes) 000285[1] 1787857601.830002 read 0x095af6 thru 0x09f126 (0x9631 bytes) 000286[0] 1787857601.867342 read 0x29dd66 thru 0x29fcb7 (0x1f52 bytes) 000292[0] 1787857601.917806 write 0x5edbd6 thru 0x5fcafd (0xef28 bytes) 000294[0] 1787857601.934727 mapwrite 0x7dbfc0 thru 0x7def8c (0x2fcd bytes) 000302[0] 1787857602.095937 trunc from 0x7def8d to 0x240a4d (0x59e53f bytes) 000304[1] 1787857602.150393 mapwrite 0x9648f0 thru 0x96b993 (0x70a4 bytes) 000308[1] 1787857602.383803 mapread 0x2a07a5 thru 0x2a2aaf (0x230b bytes) 000309[0] 1787857602.407712 mapread 0x0c92f3 thru 0x0d8697 (0xf3a5 bytes) 000315[1] 1787857602.525842 mapwrite 0x65b9b6 thru 0x65e8e4 (0x2f2f bytes) 000317[1] 1787857602.658197 mapread 0x65d412 thru 0x6692e7 (0xbed6 bytes) 000322[1] 1787857602.683712 mapwrite 0x68dae9 thru 0x68fabb (0x1fd3 bytes) 000324[1] 1787857602.812948 mapread 0x1cc22d thru 0x1ce846 (0x261a bytes) 000327[0] 1787857602.901351 mapwrite 0x79e205 thru 0x7ad02d (0xee29 bytes) 000328[0] 1787857603.069015 read 0x1c2986 thru 0x1c8816 (0x5e91 bytes) 000330[1] 1787857603.099852 mapwrite 0x432016 thru 0x440cf2 (0xecdd bytes) 000331[1] 1787857603.217939 mapwrite 0x41e4f7 thru 0x42d3be (0xeec8 bytes) 000332[0] 1787857603.326654 mapwrite 0x506010 thru 0x5081a7 (0x2198 bytes) 000333[1] 1787857603.405945 trunc from 0x96b994 to 0x69d5ea (0x2ce3a9 bytes) 000336[1] 1787857603.468602 read 0x0b78b2 thru 0x0bd372 (0x5ac1 bytes) 000337[1] 1787857603.498110 mapwrite 0x0d72e0 thru 0x0dfe24 (0x8b45 bytes) 000338[0] 1787857603.633590 write 0x9e88e9 thru 0x9ebd2f (0x3447 bytes) 000339[1] 1787857603.662992 mapread 0x625643 thru 0x626499 (0xe57 bytes) 000346[0] 1787857603.848289 mapwrite 0x387151 thru 0x38d522 (0x63d2 bytes) 000347[1] 1787857603.978396 trunc from 0x9ebd30 to 0x7a2aaf (0x249280 bytes) 000352[1] 1787857604.089932 mapwrite 0x4985f2 thru 0x4a7422 (0xee31 bytes) 000353[1] 1787857604.235693 mapread 0x2fe1c7 thru 0x30242e (0x4268 bytes) 000360[0] 1787857604.302627 mapwrite 0x973185 thru 0x98160d (0xe489 bytes) 000365[1] 1787857604.557717 mapwrite 0x8dba50 thru 0x8e460f (0x8bc0 bytes) 000367[1] 1787857604.714362 trunc from 0x98160e to 0x6a5d7b (0x2db892 bytes) 000370[1] 1787857604.827849 read 0x128888 thru 0x134044 (0xb7bd bytes) 000373[1] 1787857604.840918 read 0x518aba thru 0x523d3a (0xb281 bytes) 000375[1] 1787857604.852542 read 0x669314 thru 0x673fc3 (0xacb0 bytes) 000376[0] 1787857604.886234 trunc from 0x6a5d7c to 0x468166 (0x23dc15 bytes) 000377[1] 1787857604.954927 mapwrite 0x46c971 thru 0x475d24 (0x93b4 bytes) 000378[0] 1787857605.105404 write 0x8c241e thru 0x8c4b76 (0x2759 bytes) 000379[1] 1787857605.137429 trunc from 0x8c4b77 to 0x9575fd (0x92a87 bytes) 000380[1] 1787857605.178022 mapwrite 0x2d97b6 thru 0x2d9e0f (0x65a bytes) 000381[0] 1787857605.252923 mapread 0x8f10c6 thru 0x8fb65a (0xa595 bytes) 000384[0] 1787857605.320014 read 0x173a46 thru 0x17d401 (0x99bc bytes) 000390[1] 1787857605.378617 mapwrite 0x9b7da0 thru 0x9c4990 (0xcbf1 bytes) 000391[0] 1787857605.548255 read 0x1f2887 thru 0x1feb6c (0xc2e6 bytes) 000392[1] 1787857605.634537 mapwrite 0x96ad9d thru 0x977d1f (0xcf83 bytes) 000402[0] 1787857605.946750 read 0x4f7698 thru 0x5033a2 (0xbd0b bytes) 000407[0] 1787857606.027365 read 0x056551 thru 0x05b509 (0x4fb9 bytes) 000408[0] 1787857606.063921 mapwrite 0x6aedf3 thru 0x6b2093 (0x32a1 bytes) 000409[1] 1787857606.169301 trunc from 0x9c4991 to 0x0b75a6 (0x90d3ea bytes) 000411[1] 1787857606.250965 mapwrite 0x08fe03 thru 0x092129 (0x2327 bytes) 000413[0] 1787857606.362100 read 0x021835 thru 0x022d2f (0x14fb bytes) 000415[0] 1787857606.432967 mapwrite 0x53f7e5 thru 0x541f25 (0x2741 bytes) 000418[1] 1787857606.723721 mapwrite 0x87afbe thru 0x8853c4 (0xa407 bytes) 000419[0] 1787857606.921636 write 0x4fd2be thru 0x50b9b7 (0xe6fa bytes) 000421[1] 1787857606.972568 write 0x812a09 thru 0x8187a8 (0x5da0 bytes) 000424[0] 1787857607.028283 write 0x0cc0d7 thru 0x0d90ee (0xd018 bytes) 000425[1] 1787857607.060572 write 0x461fe7 thru 0x47166f (0xf689 bytes) 000426[1] 1787857607.120576 trunc from 0x8853c5 to 0x9aada0 (0x1259dc bytes) 000427[0] 1787857607.177508 mapwrite 0x1b4ab7 thru 0x1c1d7e (0xd2c8 bytes) 000431[0] 1787857607.396112 trunc from 0x9aada1 to 0x0702a6 (0x93aafa bytes) 000432[1] 1787857607.451948 trunc from 0x0702a7 to 0x6f48c5 (0x68461f bytes) 000433[1] 1787857607.516798 trunc from 0x6f48c6 to 0x2715c0 (0x483305 bytes) 000434[0] 1787857607.556614 write 0x232b68 thru 0x240179 (0xd612 bytes) 000435[0] 1787857607.594674 write 0x622103 thru 0x6228e5 (0x7e3 bytes) 000439[1] 1787857607.647646 read 0x212e88 thru 0x2218ef (0xea68 bytes) 000441[1] 1787857607.685721 mapwrite 0x948c35 thru 0x956188 (0xd554 bytes) 000445[0] 1787857607.887565 mapread 0x650b01 thru 0x65ee74 (0xe374 bytes) 000452[1] 1787857608.023706 mapread 0x139384 thru 0x147ff4 (0xec71 bytes) 000457[1] 1787857608.253442 write 0x39b1cc thru 0x3a0af5 (0x592a bytes) 000459[0] 1787857608.283554 mapwrite 0x31b021 thru 0x31b483 (0x463 bytes) 000462[1] 1787857608.406704 trunc from 0x956189 to 0x15f37d (0x7f6e0b bytes) 000465[0] 1787857608.513909 write 0x3ba32e thru 0x3bf209 (0x4edc bytes) 000466[0] 1787857608.554280 read 0x39d99a thru 0x39f924 (0x1f8b bytes) 000468[0] 1787857608.679240 read 0x11e5b6 thru 0x12a764 (0xc1af bytes) 000470[1] 1787857608.710472 mapread 0x309ec3 thru 0x318189 (0xe2c7 bytes) 000474[0] 1787857608.803802 write 0x9307b0 thru 0x934933 (0x4184 bytes) 000475[0] 1787857608.836130 trunc from 0x934934 to 0x4b738b (0x47d5a8 bytes) 000476[1] 1787857608.893219 mapread 0x2d58ff thru 0x2d6b7e (0x1280 bytes) 000477[0] 1787857608.943122 read 0x1914f6 thru 0x1a065d (0xf168 bytes) 000478[0] 1787857608.966490 trunc from 0x4b738c to 0x36a800 (0x14cb8b bytes) 000479[1] 1787857609.046809 read 0x037565 thru 0x040ac5 (0x9561 bytes) 000481[0] 1787857609.196066 read 0x1351db thru 0x135260 (0x86 bytes) 000482[1] 1787857609.267051 mapread 0x2e50fc thru 0x2e56a3 (0x5a8 bytes) 000486[1] 1787857609.298657 trunc from 0x36a801 to 0x6bb1cd (0x3509cd bytes) 000489[0] 1787857609.407199 write 0x2d5cae thru 0x2dc3e1 (0x6734 bytes) 000491[0] 1787857609.468301 read 0x3b4f9c thru 0x3c3f3a (0xef9f bytes) 000494[1] 1787857609.505669 read 0x045e09 thru 0x0523af (0xc5a7 bytes) 000495[0] 1787857609.554507 trunc from 0x6bb1ce to 0x48ce3f (0x22e38e bytes) 000496[0] 1787857609.626485 mapwrite 0x29f6b9 thru 0x2a398b (0x42d3 bytes) 000499[0] 1787857609.814470 write 0x66c424 thru 0x679fb3 (0xdb90 bytes) 000500[0] 1787857609.862091 mapwrite 0x12257e thru 0x12d0b8 (0xab3b bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7610 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1787857610.920339 write_OD 0x351000 thru 0x358fff (0x8000 bytes) 000003[0] 1787857611.055175 read_OD 0x2dc000 thru 0x2eafff (0xf000 bytes) 000004[0] 1787857611.075570 mapwrite 0x387000 thru 0x3933d1 (0xc3d2 bytes) 000005[1] 1787857611.230048 read_OD 0x030000 thru 0x034fff (0x5000 bytes) 000006[1] 1787857611.277555 mapread 0x1b0000 thru 0x1bb61c (0xb61d bytes) 000008[1] 1787857611.353438 write_OD 0x8e2000 thru 0x8effff (0xe000 bytes) 000011[0] 1787857611.490596 write_OD 0x594000 thru 0x59ffff (0xc000 bytes) 000012[1] 1787857611.558462 read_OD 0x330000 thru 0x338fff (0x9000 bytes) 000014[1] 1787857611.571452 mapwrite 0x15d000 thru 0x161549 (0x454a bytes) truncating to largest ever: 0x97496a 000017[1] 1787857611.672685 trunc from 0x8f0000 to 0x974969 (0x8496a bytes) 000018[1] 1787857611.692434 mapread 0x855000 thru 0x85829f (0x32a0 bytes) 000020[1] 1787857611.740896 mapread 0x0e8000 thru 0x0f37b6 (0xb7b7 bytes) 000021[1] 1787857611.790097 mapread 0x31f000 thru 0x323765 (0x4766 bytes) 000022[0] 1787857611.829677 read_OD 0x4ab000 thru 0x4adfff (0x3000 bytes) 000027[1] 1787857611.856536 mapread 0x571000 thru 0x57c8a9 (0xb8aa bytes) 000028[0] 1787857611.884593 write_OD 0x72c000 thru 0x72ffff (0x4000 bytes) 000030[1] 1787857611.935168 trunc from 0x97496a to 0x53e137 (0x436832 bytes) 000031[0] 1787857611.992804 mapwrite 0x885000 thru 0x888637 (0x3638 bytes) 000033[1] 1787857612.148687 mapread 0x7c9000 thru 0x7ced64 (0x5d65 bytes) 000034[0] 1787857612.190478 write_OD 0x29a000 thru 0x2a7fff (0xe000 bytes) 000040[0] 1787857612.240287 mapwrite 0x757000 thru 0x760940 (0x9941 bytes) 000042[1] 1787857612.356429 trunc from 0x888638 to 0x0be2cd (0x7ca36a bytes) 000043[1] 1787857612.402537 mapread 0x006000 thru 0x00b604 (0x5605 bytes) 000044[0] 1787857612.461353 mapread 0x07d000 thru 0x085e61 (0x8e62 bytes) 000045[1] 1787857612.525168 read_OD 0x06e000 thru 0x06ffff (0x2000 bytes) 000046[1] 1787857612.530847 mapwrite 0x695000 thru 0x69b17a (0x617b bytes) 000048[0] 1787857612.749718 trunc from 0x69b17b to 0x0182d6 (0x682ea4 bytes) 000049[1] 1787857612.813688 mapread 0x008000 thru 0x00c2f6 (0x42f7 bytes) 000050[0] 1787857612.860499 trunc from 0x0182d7 to 0x0d85bd (0xc02e7 bytes) truncating to largest ever: 0x9789a7 000051[0] 1787857612.895232 trunc from 0x0d85be to 0x9789a6 (0x8a03e9 bytes) 000053[1] 1787857612.968799 mapwrite 0x744000 thru 0x7498cf (0x58d0 bytes) 000056[0] 1787857613.224762 trunc from 0x9789a7 to 0x25881d (0x720189 bytes) 000057[1] 1787857613.264456 mapwrite 0x3f6000 thru 0x3ff94b (0x994c bytes) 000058[0] 1787857613.403331 trunc from 0x3ff94c to 0x84eded (0x44f4a2 bytes) 000059[0] 1787857613.451599 write_OD 0x7dc000 thru 0x7e4fff (0x9000 bytes) 000060[0] 1787857613.507621 read_OD 0x7f7000 thru 0x7fcfff (0x6000 bytes) 000061[1] 1787857613.521940 mapwrite 0x963000 thru 0x96c65a (0x965b bytes) 000063[1] 1787857613.819120 trunc from 0x96c65b to 0x808796 (0x163ec4 bytes) 000065[1] 1787857613.874941 trunc from 0x808797 to 0x37533f (0x493457 bytes) 000066[0] 1787857613.921300 mapread 0x08c000 thru 0x09a191 (0xe192 bytes) 000067[1] 1787857614.007214 mapwrite 0x16d000 thru 0x172902 (0x5903 bytes) 000068[0] 1787857614.135599 mapwrite 0x7b7000 thru 0x7c2f9d (0xbf9e bytes) 000069[1] 1787857614.246756 read_OD 0x714000 thru 0x71cfff (0x9000 bytes) 000070[1] 1787857614.279457 mapread 0x5e2000 thru 0x5eb74c (0x974d bytes) 000071[0] 1787857614.453711 read_OD 0x767000 thru 0x771fff (0xb000 bytes) 000072[0] 1787857614.465786 read_OD 0x557000 thru 0x55afff (0x4000 bytes) 000075[1] 1787857614.472703 read_OD 0x30c000 thru 0x319fff (0xe000 bytes) 000077[0] 1787857614.484722 read_OD 0x656000 thru 0x65dfff (0x8000 bytes) 000083[0] 1787857614.496908 mapwrite 0x1ae000 thru 0x1b79e5 (0x99e6 bytes) 000089[1] 1787857614.641351 write_OD 0x1ba000 thru 0x1c6fff (0xd000 bytes) 000092[1] 1787857614.695458 read_OD 0x798000 thru 0x79ffff (0x8000 bytes) 000094[0] 1787857614.702894 read_OD 0x287000 thru 0x291fff (0xb000 bytes) 000095[0] 1787857614.710176 mapread 0x22e000 thru 0x232d80 (0x4d81 bytes) 000098[0] 1787857614.724055 trunc from 0x7c2f9e to 0x69971f (0x12987e bytes) 000100[1] 1787857614.813513 read_OD 0x06f000 thru 0x07bfff (0xd000 bytes) 000101[0] 1787857614.844487 trunc from 0x699720 to 0x7f7e48 (0x15e729 bytes) 000103[1] 1787857614.885801 mapread 0x305000 thru 0x3136e5 (0xe6e6 bytes) truncating to largest ever: 0x9cc75c 000104[0] 1787857614.948858 trunc from 0x7f7e49 to 0x9cc75b (0x1d4913 bytes) 000109[0] 1787857615.094966 mapread 0x60e000 thru 0x61465d (0x665e bytes) 000111[0] 1787857615.193065 read_OD 0x40c000 thru 0x419fff (0xe000 bytes) 000117[1] 1787857615.256321 mapwrite 0x9a1000 thru 0x9a5bca (0x4bcb bytes) 000118[0] 1787857615.382678 read_OD 0x228000 thru 0x229fff (0x2000 bytes) 000120[0] 1787857615.405803 mapwrite 0x3c9000 thru 0x3d5041 (0xc042 bytes) 000121[1] 1787857615.545487 mapwrite 0x5c7000 thru 0x5cc10a (0x510b bytes) 000124[0] 1787857615.723590 mapwrite 0x737000 thru 0x73de8c (0x6e8d bytes) 000125[0] 1787857615.853362 read_OD 0x4d0000 thru 0x4dafff (0xb000 bytes) 000127[0] 1787857615.874678 mapread 0x4dc000 thru 0x4e67af (0xa7b0 bytes) 000129[1] 1787857615.899007 mapwrite 0x51c000 thru 0x5275b8 (0xb5b9 bytes) 000132[0] 1787857616.039337 mapread 0x0bc000 thru 0x0c4c05 (0x8c06 bytes) 000133[0] 1787857616.089996 mapread 0x2ff000 thru 0x30973e (0xa73f bytes) 000135[0] 1787857616.117115 read_OD 0x408000 thru 0x40efff (0x7000 bytes) 000136[1] 1787857616.128067 read_OD 0x5db000 thru 0x5ddfff (0x3000 bytes) truncating to largest ever: 0x9f3963 000138[0] 1787857616.135080 trunc from 0x9cc75c to 0x9f3962 (0x27207 bytes) 000139[0] 1787857616.180531 mapwrite 0x393000 thru 0x3a2fe5 (0xffe6 bytes) 000140[1] 1787857616.247084 trunc from 0x9f3963 to 0x13273d (0x8c1225 bytes) 000141[0] 1787857616.294340 mapread 0x0aa000 thru 0x0ab9fe (0x19ff bytes) 000143[0] 1787857616.347133 mapread 0x068000 thru 0x0743fa (0xc3fb bytes) 000145[1] 1787857616.369380 trunc from 0x13273e to 0x869147 (0x736a0a bytes) 000146[1] 1787857616.419296 mapread 0x287000 thru 0x2916ea (0xa6eb bytes) 000147[1] 1787857616.456123 mapread 0x45a000 thru 0x460f87 (0x6f88 bytes) 000148[0] 1787857616.479109 write_OD 0x6af000 thru 0x6b8fff (0xa000 bytes) 000149[0] 1787857616.580568 mapread 0x2af000 thru 0x2b324e (0x424f bytes) 000150[0] 1787857616.609659 mapread 0x83b000 thru 0x84ac84 (0xfc85 bytes) 000152[0] 1787857616.648422 trunc from 0x869148 to 0x17754b (0x6f1bfc bytes) 000154[1] 1787857616.707539 mapread 0x12e000 thru 0x13c183 (0xe184 bytes) 000155[0] 1787857616.762980 write_OD 0x060000 thru 0x06cfff (0xd000 bytes) 000156[0] 1787857616.823873 write_OD 0x23d000 thru 0x23dfff (0x1000 bytes) 000158[0] 1787857616.878524 mapread 0x15c000 thru 0x1615ec (0x55ed bytes) 000161[1] 1787857616.907774 write_OD 0x2a0000 thru 0x2a4fff (0x5000 bytes) 000163[0] 1787857616.972437 read_OD 0x08f000 thru 0x094fff (0x6000 bytes) 000164[0] 1787857617.000547 mapread 0x214000 thru 0x2148c1 (0x8c2 bytes) 000166[1] 1787857617.045056 mapread 0x1b8000 thru 0x1b89de (0x9df bytes) 000169[0] 1787857617.066018 mapread 0x18d000 thru 0x19af64 (0xdf65 bytes) 000170[0] 1787857617.066702 read_OD 0x1f8000 thru 0x203fff (0xc000 bytes) 000171[1] 1787857617.092636 write_OD 0x00f000 thru 0x016fff (0x8000 bytes) 000175[0] 1787857617.153872 mapread 0x040000 thru 0x04dda1 (0xdda2 bytes) 000176[0] 1787857617.236397 read_OD 0x089000 thru 0x091fff (0x9000 bytes) 000177[1] 1787857617.256515 write_OD 0x72b000 thru 0x733fff (0x9000 bytes) 000179[1] 1787857617.342481 write_OD 0x6f2000 thru 0x6fefff (0xd000 bytes) 000182[1] 1787857617.395465 read_OD 0x391000 thru 0x39ffff (0xf000 bytes) 000184[0] 1787857617.410045 mapread 0x259000 thru 0x261ed1 (0x8ed2 bytes) 000186[1] 1787857617.478088 mapread 0x597000 thru 0x5a1c22 (0xac23 bytes) 000191[0] 1787857617.519900 mapread 0x23e000 thru 0x23fe72 (0x1e73 bytes) 000194[0] 1787857617.524023 mapread 0x163000 thru 0x16905a (0x605b bytes) 000196[1] 1787857617.526751 mapwrite 0x031000 thru 0x03505d (0x405e bytes) 000198[0] 1787857617.718096 read_OD 0x12e000 thru 0x136fff (0x9000 bytes) 000203[0] 1787857617.764348 write_OD 0x2bd000 thru 0x2c9fff (0xd000 bytes) 000204[1] 1787857617.836150 write_OD 0x7df000 thru 0x7e8fff (0xa000 bytes) 000205[0] 1787857617.897542 read_OD 0x358000 thru 0x35efff (0x7000 bytes) 000207[0] 1787857617.910595 mapwrite 0x37d000 thru 0x3888b4 (0xb8b5 bytes) 000210[1] 1787857618.013567 write_OD 0x137000 thru 0x138fff (0x2000 bytes) 000212[0] 1787857618.107960 trunc from 0x7e9000 to 0x9a8789 (0x1bf78a bytes) 000213[1] 1787857618.152950 read_OD 0x477000 thru 0x481fff (0xb000 bytes) 000216[0] 1787857618.221651 trunc from 0x9a878a to 0x198167 (0x810622 bytes) 000217[1] 1787857618.280722 mapread 0x10b000 thru 0x113a3e (0x8a3f bytes) 000218[1] 1787857618.331988 trunc from 0x198168 to 0x151351 (0x46e16 bytes) 000219[1] 1787857618.376747 trunc from 0x151352 to 0x6d8538 (0x5871e7 bytes) 000220[1] 1787857618.410724 trunc from 0x6d8539 to 0x2571fb (0x48133d bytes) 000222[1] 1787857618.455289 mapwrite 0x6a2000 thru 0x6a2fdd (0xfde bytes) 000223[1] 1787857618.531741 mapread 0x58e000 thru 0x59a881 (0xc882 bytes) 000225[0] 1787857618.584640 mapread 0x446000 thru 0x4497b3 (0x37b4 bytes) 000227[0] 1787857618.669088 mapwrite 0x2a1000 thru 0x2a5c11 (0x4c12 bytes) 000228[1] 1787857618.806994 mapread 0x508000 thru 0x51285f (0xa860 bytes) 000231[1] 1787857618.854023 read_OD 0x42a000 thru 0x438fff (0xf000 bytes) 000233[0] 1787857618.866398 mapwrite 0x00b000 thru 0x00dffb (0x2ffc bytes) 000235[0] 1787857618.958178 read_OD 0x24a000 thru 0x256fff (0xd000 bytes) 000240[0] 1787857618.985039 write_OD 0x0f3000 thru 0x0f7fff (0x5000 bytes) 000243[0] 1787857619.105874 read_OD 0x466000 thru 0x46afff (0x5000 bytes) 000244[1] 1787857619.116732 read_OD 0x265000 thru 0x26dfff (0x9000 bytes) 000247[1] 1787857619.129642 mapwrite 0x452000 thru 0x461c36 (0xfc37 bytes) 000248[1] 1787857619.270396 trunc from 0x6a2fde to 0x3d6a73 (0x2cc56a bytes) 000249[0] 1787857619.347579 write_OD 0x2bc000 thru 0x2bffff (0x4000 bytes) 000254[1] 1787857619.443536 write_OD 0x0f3000 thru 0x0f9fff (0x7000 bytes) 000255[1] 1787857619.565346 mapwrite 0x487000 thru 0x489883 (0x2884 bytes) 000258[1] 1787857619.721461 write_OD 0x9d4000 thru 0x9d7fff (0x4000 bytes) 000259[1] 1787857619.815736 trunc from 0x9d8000 to 0x030c92 (0x9a736d bytes) 000260[1] 1787857619.853089 write_OD 0x2a8000 thru 0x2a8fff (0x1000 bytes) 000262[0] 1787857619.941249 mapread 0x173000 thru 0x176e56 (0x3e57 bytes) 000263[1] 1787857619.998561 mapread 0x114000 thru 0x119138 (0x5139 bytes) 000264[1] 1787857620.040452 trunc from 0x2a9000 to 0x0541a8 (0x254e57 bytes) 000265[0] 1787857620.077970 trunc from 0x0541a9 to 0x0d7c90 (0x83ae8 bytes) 000267[0] 1787857620.161670 mapread 0x029000 thru 0x02bd8c (0x2d8d bytes) 000269[0] 1787857620.202324 read_OD 0x07a000 thru 0x07cfff (0x3000 bytes) 000270[1] 1787857620.218259 mapwrite 0x942000 thru 0x94ba3b (0x9a3c bytes) 000271[0] 1787857620.322617 read_OD 0x333000 thru 0x335fff (0x3000 bytes) 000273[0] 1787857620.366250 read_OD 0x58d000 thru 0x58ffff (0x3000 bytes) 000274[1] 1787857620.373119 mapwrite 0x704000 thru 0x707915 (0x3916 bytes) 000275[0] 1787857620.464568 write_OD 0x07a000 thru 0x07afff (0x1000 bytes) 000276[1] 1787857620.556957 trunc from 0x94ba3c to 0x5b42cf (0x39776c bytes) 000279[1] 1787857620.672358 read_OD 0x065000 thru 0x065fff (0x1000 bytes) 000282[1] 1787857620.699348 mapwrite 0x709000 thru 0x716c3b (0xdc3c bytes) 000283[1] 1787857620.847778 write_OD 0x462000 thru 0x462fff (0x1000 bytes) 000285[1] 1787857620.900693 mapread 0x281000 thru 0x28543c (0x443d bytes) 000289[0] 1787857620.926925 mapread 0x69d000 thru 0x6aa22e (0xd22f bytes) 000290[1] 1787857620.954601 write_OD 0x5e7000 thru 0x5f1fff (0xb000 bytes) 000291[0] 1787857621.008448 trunc from 0x716c3c to 0x56ebfa (0x1a8041 bytes) 000292[0] 1787857621.055240 mapwrite 0x569000 thru 0x569710 (0x711 bytes) 000294[0] 1787857621.163775 trunc from 0x56ebfb to 0x3e82c4 (0x186936 bytes) 000296[1] 1787857621.202320 trunc from 0x3e82c5 to 0x17cdd6 (0x26b4ee bytes) 000298[0] 1787857621.261576 read_OD 0x07e000 thru 0x086fff (0x9000 bytes) 000300[0] 1787857621.295476 read_OD 0x0d1000 thru 0x0dffff (0xf000 bytes) 000301[0] 1787857621.307265 mapread 0x0fd000 thru 0x0fd6db (0x6dc bytes) 000305[0] 1787857621.344556 write_OD 0x705000 thru 0x70efff (0xa000 bytes) 000306[1] 1787857621.410705 read_OD 0x0e6000 thru 0x0eafff (0x5000 bytes) 000308[0] 1787857621.425468 write_OD 0x305000 thru 0x30afff (0x6000 bytes) 000312[1] 1787857621.492757 mapread 0x0df000 thru 0x0e5586 (0x6587 bytes) 000314[1] 1787857621.526972 read_OD 0x0eb000 thru 0x0f0fff (0x6000 bytes) 000317[1] 1787857621.547991 read_OD 0x29c000 thru 0x29ffff (0x4000 bytes) 000318[0] 1787857621.558844 mapread 0x011000 thru 0x01d4a6 (0xc4a7 bytes) 000319[1] 1787857621.591384 write_OD 0x71e000 thru 0x724fff (0x7000 bytes) 000320[0] 1787857621.652318 mapwrite 0x4eb000 thru 0x4f568b (0xa68c bytes) 000321[1] 1787857621.738911 trunc from 0x725000 to 0x70c1e9 (0x18e16 bytes) 000326[0] 1787857621.813551 mapwrite 0x8f5000 thru 0x8f64e8 (0x14e9 bytes) 000327[1] 1787857621.892433 read_OD 0x75d000 thru 0x75dfff (0x1000 bytes) 000330[1] 1787857621.923554 write_OD 0x7f9000 thru 0x807fff (0xf000 bytes) 000331[1] 1787857621.979638 mapread 0x676000 thru 0x67aeef (0x4ef0 bytes) 000333[1] 1787857622.014930 mapwrite 0x964000 thru 0x971232 (0xd233 bytes) 000340[0] 1787857622.279223 write_OD 0x5ac000 thru 0x5b9fff (0xe000 bytes) 000342[0] 1787857622.382598 read_OD 0x4bf000 thru 0x4c7fff (0x9000 bytes) 000344[0] 1787857622.401519 mapwrite 0x31a000 thru 0x32443d (0xa43e bytes) 000347[0] 1787857622.642905 trunc from 0x971233 to 0x7e350e (0x18dd24 bytes) 000349[1] 1787857622.668855 trunc from 0x7e350f to 0x335347 (0x4ae1c7 bytes) 000350[1] 1787857622.704051 mapread 0x114000 thru 0x1170c6 (0x30c7 bytes) 000352[0] 1787857622.766811 trunc from 0x335348 to 0x3ee832 (0xb94eb bytes) 000355[0] 1787857622.835251 mapread 0x394000 thru 0x39cff3 (0x8ff4 bytes) 000357[1] 1787857622.868740 write_OD 0x843000 thru 0x849fff (0x7000 bytes) 000358[0] 1787857623.010774 mapwrite 0x794000 thru 0x79b354 (0x7355 bytes) 000360[1] 1787857623.198921 trunc from 0x84a000 to 0x187f5d (0x6c20a2 bytes) 000362[1] 1787857623.267356 write_OD 0x4a5000 thru 0x4adfff (0x9000 bytes) 000364[1] 1787857623.324641 read_OD 0x0c2000 thru 0x0c9fff (0x8000 bytes) 000365[0] 1787857623.339561 mapwrite 0x59d000 thru 0x59f022 (0x2023 bytes) 000366[1] 1787857623.423915 trunc from 0x59f023 to 0x680969 (0xe1947 bytes) 000367[0] 1787857623.454067 write_OD 0x7a4000 thru 0x7affff (0xc000 bytes) 000369[0] 1787857623.543903 trunc from 0x7b0000 to 0x27c9ae (0x533651 bytes) 000370[0] 1787857623.615431 write_OD 0x4aa000 thru 0x4b8fff (0xf000 bytes) 000371[1] 1787857623.696924 trunc from 0x4b9000 to 0x9132d7 (0x45a2d8 bytes) 000372[1] 1787857623.756327 mapwrite 0x37a000 thru 0x3869b2 (0xc9b3 bytes) 000373[1] 1787857623.939290 mapread 0x072000 thru 0x077be2 (0x5be3 bytes) 000375[0] 1787857624.028613 mapread 0x7b2000 thru 0x7be0a6 (0xc0a7 bytes) 000376[0] 1787857624.106202 read_OD 0x31e000 thru 0x321fff (0x4000 bytes) 000377[1] 1787857624.128635 read_OD 0x65b000 thru 0x665fff (0xb000 bytes) 000381[0] 1787857624.142530 trunc from 0x9132d8 to 0x7fce0c (0x1164cb bytes) 000382[0] 1787857624.188885 write_OD 0x8e7000 thru 0x8f4fff (0xe000 bytes) 000388[0] 1787857624.284506 mapwrite 0x608000 thru 0x60c0c8 (0x40c9 bytes) 000389[0] 1787857624.433748 read_OD 0x047000 thru 0x048fff (0x2000 bytes) 000391[0] 1787857624.481743 read_OD 0x362000 thru 0x362fff (0x1000 bytes) 000393[1] 1787857624.501910 mapread 0x2ab000 thru 0x2b3e88 (0x8e89 bytes) 000394[0] 1787857624.527487 trunc from 0x8f5000 to 0x176571 (0x77ea8e bytes) 000395[0] 1787857624.578361 mapread 0x024000 thru 0x02c295 (0x8296 bytes) 000396[0] 1787857624.634735 mapread 0x00f000 thru 0x01e7e1 (0xf7e2 bytes) 000397[1] 1787857624.719309 mapread 0x0b0000 thru 0x0b8598 (0x8599 bytes) 000399[0] 1787857624.771852 write_OD 0x1cb000 thru 0x1ccfff (0x2000 bytes) 000400[0] 1787857624.858668 write_OD 0x5e9000 thru 0x5eafff (0x2000 bytes) 000403[0] 1787857624.929303 trunc from 0x5eb000 to 0x1fa19d (0x3f0e62 bytes) 000404[1] 1787857624.990259 mapread 0x1f4000 thru 0x1f9464 (0x5465 bytes) 000405[0] 1787857625.054311 mapwrite 0x245000 thru 0x252cf1 (0xdcf2 bytes) 000407[0] 1787857625.156235 trunc from 0x252cf2 to 0x1987ae (0xba543 bytes) 000408[1] 1787857625.172411 mapread 0x03b000 thru 0x03f147 (0x4148 bytes) 000410[1] 1787857625.221429 trunc from 0x1987af to 0x4817a3 (0x2e8ff5 bytes) 000412[0] 1787857625.280347 read_OD 0x132000 thru 0x13cfff (0xb000 bytes) 000413[0] 1787857625.320898 write_OD 0x8a5000 thru 0x8a6fff (0x2000 bytes) 000414[1] 1787857625.374628 write_OD 0x7a7000 thru 0x7b1fff (0xb000 bytes) 000416[1] 1787857625.429169 mapwrite 0x24e000 thru 0x24f561 (0x1562 bytes) 000418[0] 1787857625.529652 trunc from 0x8a7000 to 0x7f7515 (0xafaea bytes) 000420[0] 1787857625.578715 write_OD 0x4d7000 thru 0x4d8fff (0x2000 bytes) 000426[0] 1787857625.633359 write_OD 0x7af000 thru 0x7b8fff (0xa000 bytes) 000427[0] 1787857625.679641 trunc from 0x7f7516 to 0x63377b (0x1c3d9a bytes) 000429[1] 1787857625.735740 read_OD 0x3d6000 thru 0x3e3fff (0xe000 bytes) 000431[1] 1787857625.760620 mapread 0x573000 thru 0x579b1d (0x6b1e bytes) 000432[1] 1787857625.776836 mapwrite 0x433000 thru 0x438e24 (0x5e25 bytes) 000437[0] 1787857625.870510 mapwrite 0x0a2000 thru 0x0a4f74 (0x2f75 bytes) 000439[0] 1787857626.017480 mapread 0x35b000 thru 0x366da6 (0xbda7 bytes) 000440[1] 1787857626.117395 trunc from 0x63377c to 0x7e52c3 (0x1b1b48 bytes) 000441[1] 1787857626.198308 read_OD 0x14d000 thru 0x153fff (0x7000 bytes) 000442[1] 1787857626.258694 write_OD 0x136000 thru 0x13dfff (0x8000 bytes) 000444[1] 1787857626.329732 write_OD 0x158000 thru 0x15afff (0x3000 bytes) 000445[1] 1787857626.379167 mapwrite 0x589000 thru 0x595cf7 (0xccf8 bytes) 000447[1] 1787857626.470391 mapwrite 0x831000 thru 0x83ed48 (0xdd49 bytes) 000448[0] 1787857626.605374 write_OD 0x25d000 thru 0x25ffff (0x3000 bytes) 000449[1] 1787857626.650520 write_OD 0x219000 thru 0x21ffff (0x7000 bytes) 000454[1] 1787857626.723458 write_OD 0x72f000 thru 0x73cfff (0xe000 bytes) 000459[1] 1787857626.792215 read_OD 0x0a3000 thru 0x0affff (0xd000 bytes) 000460[1] 1787857626.806370 mapread 0x099000 thru 0x0a854d (0xf54e bytes) 000462[1] 1787857626.844703 read_OD 0x59f000 thru 0x5adfff (0xf000 bytes) 000464[1] 1787857626.853598 trunc from 0x83ed49 to 0x55bc3f (0x2e3109 bytes) 000466[1] 1787857626.911799 trunc from 0x55bc40 to 0x2ed092 (0x26ebad bytes) 000467[0] 1787857626.936534 read_OD 0x1a4000 thru 0x1acfff (0x9000 bytes) 000469[0] 1787857626.969672 write_OD 0x61f000 thru 0x624fff (0x6000 bytes) 000470[1] 1787857627.032334 trunc from 0x625000 to 0x416046 (0x20efb9 bytes) 000473[0] 1787857627.124168 mapwrite 0x7e2000 thru 0x7e53b2 (0x33b3 bytes) 000474[0] 1787857627.262522 read_OD 0x20f000 thru 0x21cfff (0xe000 bytes) 000475[0] 1787857627.296387 read_OD 0x7ce000 thru 0x7d6fff (0x9000 bytes) 000477[1] 1787857627.341417 read_OD 0x71d000 thru 0x72bfff (0xf000 bytes) 000481[1] 1787857627.359561 write_OD 0x989000 thru 0x995fff (0xd000 bytes) 000482[0] 1787857627.428393 trunc from 0x996000 to 0x0195af (0x97ca50 bytes) 000483[1] 1787857627.531580 write_OD 0x843000 thru 0x847fff (0x5000 bytes) 000485[1] 1787857627.621981 mapread 0x681000 thru 0x68bc3f (0xac40 bytes) 000487[0] 1787857627.647147 mapread 0x604000 thru 0x6093c9 (0x53ca bytes) 000488[0] 1787857627.663345 trunc from 0x848000 to 0x17fec4 (0x6c813b bytes) 000489[1] 1787857627.719375 mapwrite 0x730000 thru 0x733385 (0x3386 bytes) 000495[1] 1787857627.848811 mapread 0x325000 thru 0x32f13b (0xa13c bytes) 000496[1] 1787857627.890947 mapread 0x027000 thru 0x028e08 (0x1e09 bytes) 000497[0] 1787857627.906907 mapwrite 0x29c000 thru 0x2a6daa (0xadab bytes) 000498[0] 1787857628.038776 trunc from 0x733386 to 0x31651d (0x41ce68 bytes) All operations completed A-OK! PASS 16a (51s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 15:07:13 (1787857633) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.393727 s, 21.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.438042 s, 19.2 MB/s Chance of close/open is 1 in 50 Seed set to 7636 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004[0] 1787857637.070698 write 0x63ff1b thru 0x640565 (0x64b bytes) 000005[0] 1787857637.093762 write 0x5db828 thru 0x5dc061 (0x83a bytes) 000006[0] 1787857637.109625 mapwrite 0x235920 thru 0x235a97 (0x178 bytes) truncating to largest ever: 0x4b7cb8 000007[1] 1787857637.169281 trunc from 0x640566 to 0x4b7cb7 (0x1888ae bytes) 000009[0] 1787857637.247751 write 0x365ae8 thru 0x366693 (0xbac bytes) truncating to largest ever: 0x71bccd 000010[1] 1787857637.285311 trunc from 0x4b7cb8 to 0x71bccc (0x264015 bytes) 000015[1] 1787857637.377990 mapwrite 0x27de93 thru 0x27f859 (0x19c7 bytes) 000017[0] 1787857637.489943 trunc from 0x71bccd to 0x6f15ba (0x2a712 bytes) 000018[0] 1787857637.585038 mapread 0x23823d thru 0x238ecc (0xc90 bytes) truncating to largest ever: 0x819087 000020[0] 1787857637.596193 trunc from 0x6f15bb to 0x819086 (0x127acc bytes) 000021[1] 1787857637.636576 mapread 0x2c714d thru 0x2c906c (0x1f20 bytes) 000023[1] 1787857637.704618 trunc from 0x819087 to 0x6f891b (0x12076b bytes) 000024[1] 1787857637.822192 mapwrite 0x745ed0 thru 0x74757b (0x16ac bytes) 000027[1] 1787857637.972254 write 0x8174f1 thru 0x8179a7 (0x4b7 bytes) 000028[1] 1787857638.001669 read 0x3150e3 thru 0x3166d3 (0x15f1 bytes) 000033[1] 1787857638.094887 mapread 0x3b0c64 thru 0x3b0f59 (0x2f6 bytes) 000035[1] 1787857638.162681 read 0x24081e thru 0x241cd1 (0x14b4 bytes) 000036[0] 1787857638.171153 write 0x310e86 thru 0x311a74 (0xbef bytes) 000037[1] 1787857638.217995 mapwrite 0x74f4d0 thru 0x74fba2 (0x6d3 bytes) 000038[0] 1787857638.402667 trunc from 0x8179a8 to 0x818db4 (0x140d bytes) 000039[0] 1787857638.443245 mapwrite 0x27651f thru 0x2769c0 (0x4a2 bytes) 000040[1] 1787857638.521519 write 0x47dd2c thru 0x47f817 (0x1aec bytes) 000041[1] 1787857638.555168 trunc from 0x818db5 to 0x092580 (0x786834 bytes) 000044[0] 1787857638.654099 trunc from 0x092581 to 0x7f241e (0x75fe9e bytes) 000045[0] 1787857638.718189 write 0x3dd470 thru 0x3ddb3b (0x6cc bytes) 000046[1] 1787857638.739596 read 0x22134f thru 0x22165a (0x30c bytes) 000047[0] 1787857638.768664 mapread 0x0ae903 thru 0x0aeb11 (0x20f bytes) 000048[1] 1787857638.784764 trunc from 0x7f241f to 0x2fc03f (0x4f63df bytes) 000050[1] 1787857638.827751 mapread 0x2b7e95 thru 0x2b9460 (0x15cc bytes) 000052[1] 1787857638.867478 trunc from 0x2fc040 to 0x1c0e4e (0x13b1f1 bytes) 000053[1] 1787857638.899869 write 0x592436 thru 0x594136 (0x1d01 bytes) 000054[1] 1787857638.902503 read 0x198c07 thru 0x19a3fc (0x17f6 bytes) 000055[0] 1787857638.931143 read 0x5090b4 thru 0x50992d (0x87a bytes) 000061[1] 1787857638.972585 mapwrite 0x9d4cd1 thru 0x9d6af3 (0x1e23 bytes) 000063[0] 1787857639.087444 mapwrite 0x9111b4 thru 0x912f6c (0x1db9 bytes) 000064[0] 1787857639.177443 mapwrite 0x5f70c0 thru 0x5f783a (0x77b bytes) truncating to largest ever: 0x97cbfd 000066[1] 1787857639.287190 trunc from 0x9d6af4 to 0x97cbfc (0x59ef7 bytes) 000069[0] 1787857639.398049 read 0x5622ec thru 0x56351e (0x1233 bytes) 000070[1] 1787857639.426382 mapwrite 0x77c06c thru 0x77cfef (0xf84 bytes) 000072[0] 1787857639.525602 mapread 0x1e2110 thru 0x1e2f2e (0xe1f bytes) 000073[1] 1787857639.554191 write 0x7c2a90 thru 0x7c3949 (0xeba bytes) 000075[1] 1787857639.591245 write 0x885d42 thru 0x8866a4 (0x963 bytes) 000076[1] 1787857639.634651 read 0x7a7536 thru 0x7a8040 (0xb0b bytes) 000077[0] 1787857639.642183 mapwrite 0x7bef96 thru 0x7c0768 (0x17d3 bytes) 000078[1] 1787857639.768581 read 0x3d03fc thru 0x3d1132 (0xd37 bytes) 000079[1] 1787857639.842075 trunc from 0x97cbfd to 0x8d31f1 (0xa9a0b bytes) 000080[1] 1787857639.954047 trunc from 0x8d31f2 to 0x93d31a (0x6a129 bytes) 000081[1] 1787857640.024746 write 0x8e021f thru 0x8e025b (0x3d bytes) 000082[0] 1787857640.043906 mapread 0x2866d6 thru 0x287eb7 (0x17e2 bytes) 000083[0] 1787857640.130439 write 0x68df2c thru 0x68ec20 (0xcf5 bytes) 000089[0] 1787857640.389856 write 0x60a56b thru 0x60a8fc (0x392 bytes) 000090[1] 1787857640.447593 read 0x1a7995 thru 0x1a876c (0xdd8 bytes) 000091[1] 1787857640.468743 mapwrite 0x371b7b thru 0x372669 (0xaef bytes) 000092[0] 1787857640.599700 trunc from 0x93d31b to 0x52747b (0x415e9f bytes) 000094[1] 1787857640.694547 mapread 0x130399 thru 0x1318e9 (0x1551 bytes) 000095[0] 1787857640.754631 read 0x38040b thru 0x3822a3 (0x1e99 bytes) 000097[1] 1787857640.766311 trunc from 0x52747c to 0x0006dd (0x526d9e bytes) 000098[0] 1787857640.904535 trunc from 0x0006de to 0x26c2a1 (0x26bbc4 bytes) 000099[1] 1787857641.040368 trunc from 0x26c2a2 to 0x9387b6 (0x6cc515 bytes) 000100[0] 1787857641.120278 read 0x7344e9 thru 0x735001 (0xb19 bytes) 000101[0] 1787857641.189648 trunc from 0x9387b7 to 0x6e4849 (0x253f6d bytes) 000104[1] 1787857641.355150 trunc from 0x6e484a to 0x7c3be9 (0xdf3a0 bytes) 000105[1] 1787857641.443564 trunc from 0x7c3bea to 0x31b445 (0x4a87a4 bytes) 000108[0] 1787857641.590351 mapread 0x122022 thru 0x1226aa (0x689 bytes) 000109[0] 1787857641.759172 mapwrite 0x1ca97f thru 0x1cc1a9 (0x182b bytes) 000110[1] 1787857641.954311 trunc from 0x31b446 to 0x82bd6e (0x510929 bytes) 000111[0] 1787857642.006036 mapwrite 0x86ae86 thru 0x86c797 (0x1912 bytes) 000116[1] 1787857642.297288 trunc from 0x86c798 to 0x018422 (0x854375 bytes) 000119[0] 1787857642.366400 mapread 0x006bbc thru 0x007fba (0x13ff bytes) 000123[0] 1787857642.436416 write 0x6f5ed2 thru 0x6f7c47 (0x1d76 bytes) 000125[0] 1787857642.477283 mapread 0x518543 thru 0x51a112 (0x1bd0 bytes) 000131[0] 1787857642.611472 trunc from 0x6f7c48 to 0x6c527a (0x329cd bytes) 000133[1] 1787857642.722810 trunc from 0x6c527b to 0x085602 (0x63fc78 bytes) 000134[0] 1787857642.750196 mapwrite 0x2b6aff thru 0x2b85d8 (0x1ada bytes) 000135[1] 1787857642.877950 read 0x11e4eb thru 0x11edc9 (0x8df bytes) 000136[1] 1787857642.948698 mapwrite 0x61697a thru 0x6183a3 (0x1a2a bytes) 000139[0] 1787857643.143849 trunc from 0x6183a4 to 0x3b7735 (0x260c6e bytes) 000144[0] 1787857643.241359 mapread 0x282e1b thru 0x283e49 (0x102f bytes) 000146[1] 1787857643.268456 trunc from 0x3b7736 to 0x80a4c4 (0x452d8f bytes) 000148[0] 1787857643.348760 write 0x6f2ec7 thru 0x6f40a8 (0x11e2 bytes) 000150[0] 1787857643.414862 write 0x99a744 thru 0x99afee (0x8ab bytes) 000154[0] 1787857643.488546 read 0x1b6fce thru 0x1b72ac (0x2df bytes) 000162[0] 1787857643.650007 read 0x184267 thru 0x185da2 (0x1b3c bytes) 000163[1] 1787857643.662131 mapwrite 0x6a48e8 thru 0x6a68a6 (0x1fbf bytes) 000168[1] 1787857643.956926 trunc from 0x99afef to 0x96a346 (0x30ca8 bytes) 000171[1] 1787857644.076480 mapread 0x8de2ac thru 0x8dfa70 (0x17c5 bytes) 000172[1] 1787857644.123724 read 0x596c77 thru 0x598b65 (0x1eef bytes) 000175[1] 1787857644.216336 write 0x80ed5a thru 0x80f54f (0x7f6 bytes) 000176[0] 1787857644.218059 mapwrite 0x1bed12 thru 0x1bfc63 (0xf52 bytes) 000177[1] 1787857644.314793 write 0x153ddd thru 0x154dd0 (0xff4 bytes) 000180[1] 1787857644.358350 mapwrite 0x2b6f89 thru 0x2b7c1f (0xc97 bytes) 000182[0] 1787857644.496376 mapwrite 0x58bddd thru 0x58d197 (0x13bb bytes) 000184[0] 1787857644.606875 write 0x550d5d thru 0x551658 (0x8fc bytes) 000187[0] 1787857644.646631 mapwrite 0x2cbdec thru 0x2cc087 (0x29c bytes) 000191[1] 1787857644.759594 read 0x4f26a7 thru 0x4f386c (0x11c6 bytes) 000192[1] 1787857644.789643 read 0x83bb45 thru 0x83d8c6 (0x1d82 bytes) 000194[1] 1787857644.810540 write 0x00ef96 thru 0x010f6d (0x1fd8 bytes) 000195[0] 1787857644.841462 read 0x7375c9 thru 0x738a75 (0x14ad bytes) 000196[1] 1787857644.848050 read 0x2931d3 thru 0x293b93 (0x9c1 bytes) 000200[1] 1787857644.877049 mapread 0x122f36 thru 0x12316a (0x235 bytes) 000201[1] 1787857644.925366 mapwrite 0x6f64ad thru 0x6f736c (0xec0 bytes) 000203[1] 1787857645.019016 mapwrite 0x8f60ec thru 0x8f6174 (0x89 bytes) 000205[1] 1787857645.071153 trunc from 0x96a347 to 0x93d0a0 (0x2d2a6 bytes) 000206[1] 1787857645.094588 mapwrite 0x2af320 thru 0x2b128f (0x1f70 bytes) 000211[0] 1787857645.215471 mapread 0x10fbc3 thru 0x11057b (0x9b9 bytes) 000212[1] 1787857645.323273 read 0x04dbf4 thru 0x04e9a0 (0xdad bytes) 000213[1] 1787857645.341685 mapwrite 0x3d3ab2 thru 0x3d46e3 (0xc32 bytes) 000214[1] 1787857645.569550 mapread 0x584bf7 thru 0x5859c4 (0xdce bytes) 000215[0] 1787857645.612223 write 0x866d90 thru 0x867ab9 (0xd2a bytes) 000216[0] 1787857645.675601 mapwrite 0x1c3584 thru 0x1c4e7e (0x18fb bytes) 000217[0] 1787857645.745119 write 0x3db740 thru 0x3dd18f (0x1a50 bytes) 000219[0] 1787857645.790359 trunc from 0x93d0a1 to 0x2f88bd (0x6447e3 bytes) 000221[0] 1787857645.854344 read 0x090aff thru 0x0914c1 (0x9c3 bytes) 000223[1] 1787857645.935134 mapwrite 0x90a164 thru 0x90b090 (0xf2d bytes) 000227[1] 1787857646.174956 read 0x79f924 thru 0x7a127e (0x195b bytes) 000230[0] 1787857646.183587 trunc from 0x90b091 to 0x72607b (0x1e5015 bytes) 000231[0] 1787857646.310117 write 0x2aca74 thru 0x2acc93 (0x220 bytes) 000234[1] 1787857646.373909 mapread 0x167ba4 thru 0x168f05 (0x1362 bytes) 000235[0] 1787857646.429180 trunc from 0x72607c to 0x2f58fc (0x43077f bytes) 000236[1] 1787857646.508728 trunc from 0x2f58fd to 0x3ba58c (0xc4c90 bytes) 000237[0] 1787857646.598199 trunc from 0x3ba58d to 0x3bccea (0x275e bytes) 000238[0] 1787857646.693604 read 0x393f08 thru 0x3957ab (0x18a4 bytes) 000241[1] 1787857646.910050 mapread 0x2b62e5 thru 0x2b80a4 (0x1dc0 bytes) 000245[1] 1787857647.029075 write 0x41d011 thru 0x41dee4 (0xed4 bytes) 000246[1] 1787857647.043416 mapwrite 0x69198f thru 0x692f4b (0x15bd bytes) 000248[0] 1787857647.164537 read 0x4daaf7 thru 0x4dca86 (0x1f90 bytes) 000250[0] 1787857647.200167 mapwrite 0x48ec0a thru 0x490661 (0x1a58 bytes) 000251[0] 1787857647.351916 write 0x855053 thru 0x856797 (0x1745 bytes) 000252[1] 1787857647.392846 mapread 0x5b4bc0 thru 0x5b4c71 (0xb2 bytes) 000255[0] 1787857647.489104 write 0x086009 thru 0x0867a1 (0x799 bytes) 000258[0] 1787857647.548153 read 0x166672 thru 0x1681d1 (0x1b60 bytes) 000260[0] 1787857647.570575 write 0x2a4d78 thru 0x2a596d (0xbf6 bytes) 000261[1] 1787857647.581088 mapread 0x7e0b1a thru 0x7e2705 (0x1bec bytes) 000268[0] 1787857647.833143 mapwrite 0x1038ab thru 0x103c0f (0x365 bytes) 000269[1] 1787857647.971673 mapwrite 0x0174f0 thru 0x018aa3 (0x15b4 bytes) 000271[0] 1787857648.230089 mapread 0x01371a thru 0x014538 (0xe1f bytes) 000272[1] 1787857648.341125 trunc from 0x856798 to 0x33de29 (0x51896e bytes) 000276[1] 1787857648.480014 mapread 0x028926 thru 0x0294c3 (0xb9e bytes) 000280[0] 1787857648.587317 mapwrite 0x794700 thru 0x796138 (0x1a39 bytes) 000281[1] 1787857648.698240 mapread 0x55b421 thru 0x55cf8e (0x1b6e bytes) 000283[1] 1787857648.756959 trunc from 0x796139 to 0x797503 (0x13cb bytes) 000284[1] 1787857648.824952 write 0x78f2af thru 0x790a2e (0x1780 bytes) 000285[0] 1787857648.843323 mapread 0x0a4c9e thru 0x0a5ae2 (0xe45 bytes) 000291[0] 1787857648.907025 trunc from 0x797504 to 0x861709 (0xca206 bytes) 000292[0] 1787857649.013010 mapread 0x5f3417 thru 0x5f4ba3 (0x178d bytes) 000293[1] 1787857649.031499 trunc from 0x86170a to 0x804516 (0x5d1f3 bytes) 000295[1] 1787857649.076359 mapread 0x50057a thru 0x502319 (0x1da0 bytes) 000297[1] 1787857649.140499 trunc from 0x804517 to 0x90cb5d (0x108647 bytes) 000298[1] 1787857649.198968 mapwrite 0x00c7e9 thru 0x00cb59 (0x371 bytes) 000299[0] 1787857649.263480 mapwrite 0x67f193 thru 0x67f865 (0x6d3 bytes) 000305[0] 1787857649.464396 trunc from 0x90cb5e to 0x8bddbe (0x4ed9f bytes) 000306[0] 1787857649.525053 mapwrite 0x8e9898 thru 0x8eb758 (0x1ec1 bytes) 000307[0] 1787857649.649678 read 0x4dc4a8 thru 0x4dc86e (0x3c7 bytes) 000308[0] 1787857649.659463 trunc from 0x8eb759 to 0x29f290 (0x64c4c8 bytes) 000317[0] 1787857649.799845 trunc from 0x29f291 to 0x3c5aa5 (0x126815 bytes) 000320[1] 1787857649.848467 mapread 0x3a9101 thru 0x3a9aec (0x9ec bytes) 000321[0] 1787857649.891638 trunc from 0x3c5aa6 to 0x1a86a2 (0x21d403 bytes) 000323[1] 1787857649.952230 mapwrite 0x6bc639 thru 0x6bc703 (0xcb bytes) 000324[1] 1787857650.056790 write 0x9e6312 thru 0x9e68c4 (0x5b3 bytes) 000326[0] 1787857650.083087 trunc from 0x9e68c5 to 0x388ca8 (0x65dc1c bytes) 000328[0] 1787857650.163725 mapwrite 0x14586d thru 0x14697f (0x1113 bytes) 000332[0] 1787857650.370014 read 0x0ce7ec thru 0x0d01ca (0x19df bytes) 000333[0] 1787857650.423707 trunc from 0x388ca9 to 0x125c31 (0x263077 bytes) 000335[0] 1787857650.584912 trunc from 0x125c32 to 0x314eb5 (0x1ef284 bytes) 000337[1] 1787857650.742077 mapread 0x244047 thru 0x245e87 (0x1e41 bytes) 000338[1] 1787857650.778064 trunc from 0x314eb6 to 0x43f5f5 (0x12a740 bytes) 000340[0] 1787857650.838627 write 0x8ff58e thru 0x9005a0 (0x1013 bytes) 000344[0] 1787857650.972925 mapwrite 0x0a5497 thru 0x0a5948 (0x4b2 bytes) 000351[0] 1787857651.202753 write 0x30aeed thru 0x30c0c7 (0x11db bytes) 000353[0] 1787857651.246760 mapwrite 0x58540a thru 0x585518 (0x10f bytes) 000354[0] 1787857651.341402 write 0x92ce7b thru 0x92d8a7 (0xa2d bytes) 000355[1] 1787857651.343426 trunc from 0x92d8a8 to 0x42b588 (0x50231f bytes) 000356[0] 1787857651.391642 mapwrite 0x37c3f1 thru 0x37e1f1 (0x1e01 bytes) 000357[1] 1787857651.488485 read 0x36ddcb thru 0x36eaa6 (0xcdc bytes) 000358[0] 1787857651.548349 mapwrite 0x3eb6cb thru 0x3ec865 (0x119b bytes) 000359[1] 1787857651.646687 trunc from 0x42b589 to 0x107d30 (0x323858 bytes) 000361[1] 1787857651.733306 trunc from 0x107d31 to 0x4f6561 (0x3ee831 bytes) 000362[1] 1787857651.780395 mapwrite 0x5204c4 thru 0x5209f7 (0x534 bytes) 000363[0] 1787857651.838777 trunc from 0x5209f8 to 0x017260 (0x509797 bytes) 000364[1] 1787857651.910235 mapread 0x015f04 thru 0x016041 (0x13e bytes) truncating to largest ever: 0x9e6f2b 000365[0] 1787857651.968228 trunc from 0x017261 to 0x9e6f2a (0x9cfcca bytes) 000366[1] 1787857652.024222 mapwrite 0x12ce66 thru 0x12e4b9 (0x1654 bytes) 000369[1] 1787857652.258202 write 0x1bed28 thru 0x1c02e2 (0x15bb bytes) 000370[0] 1787857652.302509 mapwrite 0x705603 thru 0x7075c6 (0x1fc4 bytes) 000371[1] 1787857652.507891 trunc from 0x9e6f2b to 0x933a2a (0xb3500 bytes) 000372[1] 1787857652.590632 mapread 0x89061b thru 0x891d25 (0x170b bytes) 000373[0] 1787857652.629192 mapread 0x183dbf thru 0x1844e3 (0x725 bytes) 000374[1] 1787857652.694890 mapwrite 0x541f63 thru 0x542fab (0x1049 bytes) 000379[0] 1787857652.777712 mapwrite 0x5319fc thru 0x533142 (0x1747 bytes) 000382[1] 1787857652.891203 write 0x831906 thru 0x832119 (0x814 bytes) 000388[1] 1787857653.177452 write 0x0354d4 thru 0x036193 (0xcc0 bytes) 000389[1] 1787857653.225779 mapread 0x4f0758 thru 0x4f15d0 (0xe79 bytes) 000393[0] 1787857653.271581 trunc from 0x933a2b to 0x59d663 (0x3963c7 bytes) 000394[1] 1787857653.351731 read 0x181f1c thru 0x182ee1 (0xfc6 bytes) 000395[1] 1787857653.373001 read 0x00f0db thru 0x0108ef (0x1815 bytes) 000396[0] 1787857653.417711 trunc from 0x59d664 to 0x4ca90f (0xd2d54 bytes) 000397[0] 1787857653.484013 trunc from 0x4ca910 to 0x372d4a (0x157bc5 bytes) 000400[1] 1787857653.542288 read 0x13db11 thru 0x13f81c (0x1d0c bytes) 000402[0] 1787857653.669304 trunc from 0x372d4b to 0x870448 (0x4fd6fe bytes) 000403[1] 1787857653.790359 trunc from 0x870449 to 0x981b58 (0x111710 bytes) 000405[1] 1787857653.885009 read 0x1175ae thru 0x1176a8 (0xfb bytes) 000407[1] 1787857653.915389 read 0x25e741 thru 0x25fbc3 (0x1483 bytes) 000410[0] 1787857653.984380 read 0x8e6ce2 thru 0x8e861e (0x193d bytes) 000411[0] 1787857654.009419 write 0x70a877 thru 0x70bebc (0x1646 bytes) 000414[1] 1787857654.047253 mapread 0x8f1a76 thru 0x8f1f3e (0x4c9 bytes) 000419[1] 1787857654.094145 mapread 0x38d4c8 thru 0x38e384 (0xebd bytes) 000421[0] 1787857654.126183 trunc from 0x981b59 to 0x504dfa (0x47cd5e bytes) 000423[1] 1787857654.181674 mapwrite 0x3db7af thru 0x3dcb5c (0x13ae bytes) 000424[1] 1787857654.287568 read 0x2e122c thru 0x2e29c9 (0x179e bytes) 000425[0] 1787857654.307724 write 0x32e4b6 thru 0x32e916 (0x461 bytes) 000426[1] 1787857654.353177 mapwrite 0x9fe7f2 thru 0x9fffff (0x180e bytes) 000427[0] 1787857654.588097 write 0x0361c2 thru 0x036718 (0x557 bytes) 000428[1] 1787857654.644334 mapread 0x610507 thru 0x6116ea (0x11e4 bytes) 000432[1] 1787857654.733868 trunc from 0xa00000 to 0x8fec2f (0x1013d0 bytes) 000435[0] 1787857654.854581 mapwrite 0x6a182d thru 0x6a3226 (0x19fa bytes) 000436[0] 1787857654.965899 trunc from 0x8fec30 to 0x28553a (0x6796f5 bytes) 000438[0] 1787857655.034796 mapwrite 0x96413e thru 0x96550a (0x13cd bytes) 000439[1] 1787857655.168379 trunc from 0x96550b to 0x248916 (0x71cbf4 bytes) 000441[0] 1787857655.316393 read 0x1d5743 thru 0x1d6243 (0xb01 bytes) 000444[0] 1787857655.458646 mapwrite 0x9e320d thru 0x9e466b (0x145f bytes) 000447[1] 1787857655.902369 trunc from 0x9e466c to 0x2bdbee (0x726a7d bytes) 000448[0] 1787857656.094983 read 0x213928 thru 0x2143ad (0xa86 bytes) 000450[1] 1787857656.251446 mapread 0x072fc3 thru 0x073def (0xe2d bytes) 000452[1] 1787857656.283563 mapread 0x014c6b thru 0x0165a8 (0x193e bytes) 000456[0] 1787857656.459026 mapwrite 0x05a1cf thru 0x05b067 (0xe99 bytes) 000457[1] 1787857656.651293 trunc from 0x2bdbef to 0x9bc381 (0x6fe793 bytes) 000459[1] 1787857656.759918 write 0x781d8c thru 0x782c61 (0xed6 bytes) 000461[1] 1787857656.836276 mapread 0x30595c thru 0x307713 (0x1db8 bytes) 000462[0] 1787857656.871131 write 0x903463 thru 0x9042c5 (0xe63 bytes) 000463[1] 1787857657.000673 mapread 0x93f3b9 thru 0x941358 (0x1fa0 bytes) 000464[0] 1787857657.152684 read 0x5f9f03 thru 0x5fa731 (0x82f bytes) 000466[1] 1787857657.211117 write 0x50e902 thru 0x50eed5 (0x5d4 bytes) 000467[1] 1787857657.242414 mapwrite 0x6ebaf9 thru 0x6ed8f0 (0x1df8 bytes) 000470[0] 1787857657.354528 mapwrite 0x83bba1 thru 0x83be29 (0x289 bytes) 000474[1] 1787857657.469152 trunc from 0x9bc382 to 0x26df44 (0x74e43d bytes) 000476[0] 1787857657.535355 write 0x045363 thru 0x04732c (0x1fca bytes) 000478[1] 1787857657.582807 mapread 0x13d530 thru 0x13d5f8 (0xc9 bytes) 000479[1] 1787857657.627678 write 0x1abebe thru 0x1ad845 (0x1988 bytes) 000480[1] 1787857657.677658 read 0x159a36 thru 0x15a132 (0x6fd bytes) 000481[1] 1787857657.800188 mapwrite 0x7b0b6f thru 0x7b12fa (0x78c bytes) 000482[0] 1787857658.093452 trunc from 0x7b12fb to 0x2a72b2 (0x50a048 bytes) 000483[0] 1787857658.161616 mapwrite 0x6f4bfa thru 0x6f4c00 (0x7 bytes) 000484[1] 1787857658.225754 mapwrite 0x8ebe64 thru 0x8edb60 (0x1cfd bytes) 000485[1] 1787857658.370550 mapread 0x5e9efe thru 0x5ea15b (0x25e bytes) 000489[1] 1787857658.407587 trunc from 0x8edb61 to 0x75dee1 (0x18fc7f bytes) 000490[1] 1787857658.434466 mapwrite 0x80a91e thru 0x80aa78 (0x15b bytes) 000491[0] 1787857658.512923 read 0x65d5ed thru 0x65d6b3 (0xc7 bytes) 000492[1] 1787857658.539734 read 0x542468 thru 0x5428af (0x448 bytes) 000494[1] 1787857658.557744 mapwrite 0x9cbe71 thru 0x9ccda2 (0xf32 bytes) 000496[1] 1787857658.671649 mapread 0x60e2e9 thru 0x60f12e (0xe46 bytes) 000498[0] 1787857658.684209 read 0x42f715 thru 0x42ff73 (0x85f bytes) 000499[0] 1787857658.704113 write 0x3cbe8d thru 0x3cc9b8 (0xb2c bytes) All operations completed A-OK! PASS 16b (31s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 15:07:44 (1787857664) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 15:07:50 (1787857670) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.311916 s, 26.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.389103 s, 21.6 MB/s ldlm.namespaces.MGC192.168.204.132@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9de003eca000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9de010b9d800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9de010b9d800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9de010b9d800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.137 s, 9.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.86081 s, 27.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.90725 s, 36.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.73129 s, 15.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.81147 s, 11.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.52791 s, 29.7 MB/s PASS 16d (51s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 15:08:40 (1787857720) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.330416 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118813 s, 8.8 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 15:08:48 (1787857728) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 15:09:17 (1787857757) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 15:09:46 (1787857786) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (8s) == sanityn test 16i: read after truncate file ============ 15:09:54 (1787857794) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 15:10:01 (1787857801) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.88359 s, 13.9 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.40077 s, 18.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.31189 s, 11.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51265 s, 17.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99965 s, 13.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.91275 s, 13.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53802 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.15064 s, 12.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.18361 s, 12.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.86073 s, 9.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.06789 s, 8.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57634 s, 16.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67745 s, 15.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41122 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.46985 s, 10.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72878 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.99179 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.07617 s, 8.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16259 s, 8.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (36s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 15:10:37 (1787857837) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Aug 27 15:11:38 EDT 2026 + fsx 2 done: rc=0 Thu Aug 27 15:11:38 EDT 2026 + fsx 3 done: rc=0 Thu Aug 27 15:11:38 EDT 2026 + fsx 4 done: rc=0 Thu Aug 27 15:11:38 EDT 2026 + fsx 5 done: rc=0 Thu Aug 27 15:11:38 EDT 2026 PASS 16k (72s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 15:11:49 (1787857909) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 15:12:02 (1787857922) mmap test1: basic mmap operation (PASS, 0.18469s) mmap test2: MAP_PRIVATE not write back (PASS, 0.178s) mmap test3: concurrent mmap ops on two nodes (PASS, 14.597s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6203s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.21525s) mmap test6: check mmap write/read content on two nodes (PASS, 0.22696s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3749s) mmap test11: multi-thread shared mmap (PASS, 1.1558s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1299s) PASS 18 (33s) == sanityn test 19: test concurrent uncached read races ========================================================================= 15:12:35 (1787857955) oleg432-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 15:12:40 (1787857960) PASS 20 (8s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 15:12:49 (1787857969) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (7s) == sanityn test 23: others should see updated atime while another read============================================================== 15:12:56 (1787857976) now is 1787857979 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 new atime is 1787858040 PASS 23 (72s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 15:14:08 (1787858048) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 15:14:16 (1787858056) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 15:14:23 (1787858063) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 15:14:29 (1787858069) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 15:14:32 (1787858072) utime: good mknod times 1787858073 <= 1787858074 <= 1787858074 for /mnt/lustre/f26a utime: good mknod times 1787858073 <= 1787858074 <= 1787858074 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 15:14:40 (1787858080) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 15:14:49 (1787858089) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-27 15:14:52.000000000 -0400 Birth: 2026-08-27 15:14:52.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-08-27 15:14:52.000000000 -0400 Birth: 2026-08-27 15:14:52.000000000 -0400 PASS 26c (11s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 15:15:02 (1787858102) 0 PASS 30 (10s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 15:15:12 (1787858112) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 15:15:20 (1787858120) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.132@tcp:/lustre 7542784 38912 7497728 1% /mnt/lustre2 PASS 31b (23s) == sanityn test 31r: open-rename(replace) race =========== 15:15:43 (1787858143) -rw-r--r-- 1 root root 0 Aug 27 15:15 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 15:15:53 (1787858153) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43158 Terminated tail -f $tfile PASS 31s (9s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 15:16:02 (1787858162) PASS 31t (10s) == sanityn test 32b: lockless i/o ======================== 15:16:12 (1787858172) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (5s) == sanityn test 32c: contention detection ================ 15:16:17 (1787858177) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.27065 s, 41.3 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.49886 s, 35.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.30166 s, 22.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.88729 s, 18.2 MB/s contention_seconds 4, contention_events 25 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.47999 s, 35.4 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.72644 s, 30.4 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.3197 s, 39.7 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31877 s, 39.8 MB/s PASS 32c (30s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 15:16:49 (1787858209) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (5s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 15:16:55 (1787858215) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (5s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 15:16:59 (1787858219) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (5s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 15:17:04 (1787858224) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 15:17:08 (1787858228) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9de003eca000.ost_server_uuid in IDLE state after 0 sec PASS 34 (144s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 15:19:33 (1787858373) fail_loc=0x80000317 Race attempt 0 Wait for 50470 50497 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1855: 50470 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.132@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre 192.168.204.132@tcp:/lustre 7.2G 38M 7.2G 1% /mnt/lustre2 - open/close 641 (time 1787858390.63 total 10.01 last 64.02) - open/close 1228 (time 1787858400.64 total 20.02 last 58.62) - open/close 1772 (time 1787858410.64 total 30.03 last 54.39) - open/close 2410 (time 1787858420.65 total 40.03 last 63.76) - open/close 3013 (time 1787858430.66 total 50.05 last 60.22) - open/close 3751 (time 1787858440.67 total 60.05 last 73.75) total: 4000 open/close in 66.06 seconds: 60.55 ops/second PASS 35 (75s) == sanityn test 36: handle ESTALE/open-unlink correctly == 15:20:47 (1787858447) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.976184 s, 53.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7499776) after_dd(7427072) after(7478272) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.645625 s, 81.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7478272) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.76396 s, 68.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7489536) after_dd(7431168) after(7482368) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.914258 s, 57.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(3) *** before(7482368) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.813604 s, 64.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.592898 s, 88.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7438336) after(7489536) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.641943 s, 81.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7489536) after_dd(7427072) after(7499776) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.695209 s, 75.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7499776) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.626132 s, 83.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7490560) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.541388 s, 96.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7492608) after_dd(7434240) after(7485440) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.631742 s, 83.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7485440) after_dd(7439360) after(7490560) PASS 36 (300s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 15:25:47 (1787858747) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 - create 1138 (time 1787858761.84 total 10.01 last 113.65) - create 1975 (time 1787858771.85 total 20.03 last 83.60) - create 3335 (time 1787858781.85 total 30.03 last 135.92) - create 4531 (time 1787858791.86 total 40.04 last 119.52) - create 5876 (time 1787858801.87 total 50.04 last 134.42) - create 7263 (time 1787858811.87 total 60.05 last 138.69) - create 8439 (time 1787858821.87 total 70.05 last 117.56) - create 9752 (time 1787858831.88 total 80.05 last 131.22) total: 10000 create in 81.84 seconds: 122.20 ops/second PASS 37 (93s) == sanityn test 39a: file mtime does not change after rename ========================================================== 15:27:21 (1787858841) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 15:27:28 (1787858848) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 15:27:36 (1787858856) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 15:27:44 (1787858864) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 15:27:54 (1787858874) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 15:28:11 (1787858891) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 15:28:27 (1787858907) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (15s) == sanityn test 40e: pdirops: rename and others ======================================================================== 15:28:42 (1787858922) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (15s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 15:28:57 (1787858937) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (13s) == sanityn test 41b: pdirops: create vs create ======================================================================== 15:29:10 (1787858950) open: File exists Conflict PASS 41b (15s) == sanityn test 41c: pdirops: create vs link ======================================================================== 15:29:26 (1787858966) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 15:29:40 (1787858980) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 15:29:54 (1787858994) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 15:30:07 (1787859007) Conflict PASS 41f (14s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 15:30:21 (1787859021) Conflict PASS 41g (14s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 15:30:36 (1787859036) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (13s) == sanityn test 41i: reint_open: create vs create ======== 15:30:48 (1787859048) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1077s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 15:48:45 (1787860125) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (14s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 15:48:59 (1787860139) open: File exists Conflict PASS 42b (14s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 15:49:14 (1787860154) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 15:49:29 (1787860169) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 15:49:44 (1787860184) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 15:49:57 (1787860197) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 15:50:11 (1787860211) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 15:50:26 (1787860226) Conflict PASS 42h (14s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 15:50:40 (1787860240) PASS 43a (63s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 15:51:43 (1787860303) Conflict PASS 43b (13s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 15:51:57 (1787860317) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 15:52:11 (1787860331) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 15:52:24 (1787860344) Conflict PASS 43e (14s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 15:52:38 (1787860358) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (15s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 15:52:53 (1787860373) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 15:53:06 (1787860386) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 15:53:21 (1787860401) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 15:53:24 (1787860404) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (131s) == sanityn test 43k: unlink vs create ==================== 15:55:35 (1787860535) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1223s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 16:15:58 (1787861758) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Aug 27 16:16:07 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 16:16:13 (1787861773) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 16:16:27 (1787861787) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 16:16:41 (1787861801) Conflict PASS 44d (14s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 16:16:55 (1787861815) Conflict PASS 44e (13s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 16:17:09 (1787861829) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 16:17:23 (1787861843) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 16:17:38 (1787861858) Conflict PASS 44h (17s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 16:17:55 (1787861875) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 16:17:59 (1787861879) PASS 45a (130s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 16:20:10 (1787862010) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 16:20:25 (1787862025) Conflict PASS 45c (16s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 16:20:42 (1787862042) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (17s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 16:20:59 (1787862059) Conflict PASS 45e (15s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 16:21:15 (1787862075) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (16s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 16:21:30 (1787862090) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (20s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 16:21:50 (1787862110) Conflict PASS 45h (16s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 16:22:07 (1787862127) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 16:22:11 (1787862131) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1247s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 16:42:58 (1787863378) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 16:43:13 (1787863393) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 16:43:26 (1787863406) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (13s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 16:43:40 (1787863420) Conflict PASS 46d (12s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 16:43:52 (1787863432) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 16:44:05 (1787863445) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 16:44:17 (1787863457) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 16:44:32 (1787863472) Conflict PASS 46h (17s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 16:44:48 (1787863488) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 16:44:52 (1787863492) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 16:44:55 (1787863495) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 16:44:58 (1787863498) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 16:45:02 (1787863502) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 16:45:05 (1787863505) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 16:45:09 (1787863509) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 16:45:12 (1787863512) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 16:45:15 (1787863515) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.146788 s, 69.8 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 16:45:28 (1787863528) 0+1 records in 0+1 records out 158 bytes copied, 0.0371093 s, 4.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218434) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 16:45:37 (1787863537) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0101893 s, 100 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612921 s, 17.1 MB/s 3145728 PASS 51b (19s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 16:45:56 (1787863556) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0312029 s, 32.8 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 16:46:08 (1787863568) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268386 s, 39.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 16:46:16 (1787863576) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (8s) == sanityn test 54: rename locking ======================= 16:46:24 (1787863584) llite.lustre-ffff9de003eca000.xattr_cache=1 llite.lustre-ffff9de010b9d800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:25.000000000 -0400 Modify: 2026-08-27 16:46:25.000000000 -0400 Change: 2026-08-27 16:46:25.000000000 -0400 Birth: 2026-08-27 16:46:26.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:32.000000000 -0400 Modify: 2026-08-27 16:46:33.000000000 -0400 Change: 2026-08-27 16:46:33.000000000 -0400 Birth: 2026-08-27 16:46:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:38.000000000 -0400 Modify: 2026-08-27 16:46:38.000000000 -0400 Change: 2026-08-27 16:46:38.000000000 -0400 Birth: 2026-08-27 16:46:38.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:38.000000000 -0400 Modify: 2026-08-27 16:46:39.000000000 -0400 Change: 2026-08-27 16:46:39.000000000 -0400 Birth: 2026-08-27 16:46:38.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:44.000000000 -0400 Modify: 2026-08-27 16:46:45.000000000 -0400 Change: 2026-08-27 16:46:45.000000000 -0400 Birth: 2026-08-27 16:46:44.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:46:44.000000000 -0400 Modify: 2026-08-27 16:46:45.000000000 -0400 Change: 2026-08-27 16:46:45.000000000 -0400 Birth: 2026-08-27 16:46:44.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (31s) == sanityn test 55a: rename vs unlink target dir ========= 16:46:55 (1787863615) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (12s) == sanityn test 55b: rename vs unlink source dir ========= 16:47:07 (1787863627) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (11s) == sanityn test 55c: rename vs unlink orphan target dir == 16:47:18 (1787863638) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (17s) == sanityn test 55d: rename file vs link ================= 16:47:35 (1787863655) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (14s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 16:47:49 (1787863669) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 16:47:52 (1787863672) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 16:48:12 (1787863692) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (22s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 16:48:34 (1787863714) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1787863716 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.744379 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (89s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 16:50:03 (1787863803) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1787863806 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.29866 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (105s) == sanityn test 60: Verify data_version behaviour ======== 16:51:49 (1787863909) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 16:51:58 (1787863918) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 16:52:08 (1787863928) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 16:52:18 (1787863938) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0330007 s, 2.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (6s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 16:52:23 (1787863943) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0162233 s, 4.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 16:52:27 (1787863947) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (5s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 16:52:32 (1787863952) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0354333 s, 1.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (3s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 16:52:36 (1787863956) llite.lustre-ffff9de003eca000.xattr_cache=1 llite.lustre-ffff9de010b9d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (10s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 16:52:45 (1787863965) llite.lustre-ffff9de003eca000.xattr_cache=1 llite.lustre-ffff9de010b9d800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (9s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 16:52:54 (1787863974) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0208066 s, 49.2 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00952548 s, 108 kB/s 231229: get lock2 231234: get lock1 231234: done 231234: get lock2 231229: done 231229: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 231229: cannot get lock1: Resource deadlock avoided 231234: done 231234: exit rc=0 231229: done 231229: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 16:53:06 (1787863986) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0670965 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235453 s, 44.5 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 231816 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (13s) == sanityn test 76: Verify MDT open_files listing ======== 16:53:19 (1787863999) opening files: ........ closing files: ........ PASS 76 (53s) == sanityn test 77a: check FIFO NRS policy =============== 16:54:13 (1787864053) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.23034 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705422 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563205 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102326 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959466 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109233 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973925 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136888 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440017 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751887 s, 13.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164702 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200798 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189778 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249875 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237525 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192994 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245498 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188226 s, 55.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460843 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488589 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00609813 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351494 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565045 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565231 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559325 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459022 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538988 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776922 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118218 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126587 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00615362 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852395 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221678 s, 47.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (9s) == sanityn test 77b: check CRR-N NRS policy ============== 16:54:22 (1787864062) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.247171 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656018 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494614 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088812 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123902 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180864 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159995 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207084 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909309 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18978 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168825 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202572 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156952 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234016 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257517 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165524 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155661 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742977 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438467 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113177 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444704 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1041 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117418 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250147 s, 419 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706589 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734673 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630584 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639142 s, 16.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404042 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277403 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386677 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605325 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529601 s, 19.8 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.258511 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239104 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293344 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437049 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524253 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331023 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718666 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474094 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762965 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499365 s, 21.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156646 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177665 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912821 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159276 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156238 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194175 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150092 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333317 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746378 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579953 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825564 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689875 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595224 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589396 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103137 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11176 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741437 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105867 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764887 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917899 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119479 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13089 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946857 s, 11.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 16:54:34 (1787864074) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.209089 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441938 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239593 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475896 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259659 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698507 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904001 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111254 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142471 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160983 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164479 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106304 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179651 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197267 s, 5.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200028 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202053 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305391 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127071 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479277 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354302 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186978 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105906 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144966 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10003 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535602 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140019 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627636 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058636 s, 17.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778939 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159816 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082462 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833743 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17641 s, 5.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270785 s, 62.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144521 s, 72.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320405 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471088 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579336 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0837964 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725781 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294288 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639917 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741488 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679881 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846822 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249697 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680663 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216629 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592373 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636243 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671815 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630237 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681997 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782643 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714817 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769294 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446092 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799828 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517432 s, 20.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985901 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133516 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656291 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670176 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659555 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809672 s, 13.0 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.212174 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278636 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808271 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996415 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121594 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950948 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118712 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905877 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092146 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163613 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116502 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182349 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145061 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211859 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233905 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21325 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281224 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00230281 s, 455 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00316167 s, 332 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939191 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143191 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00415188 s, 253 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00317608 s, 330 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118341 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810437 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442672 s, 23.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188579 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123498 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306142 s, 34.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420137 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264639 s, 39.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288862 s, 36.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253827 s, 41.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (14s) == sanityn test 77d: check TRR nrs policy ================ 16:54:49 (1787864089) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.212519 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295964 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343648 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963379 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103639 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830987 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562079 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554855 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337402 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806529 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514933 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492344 s, 21.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197579 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23335 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203165 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157944 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126736 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263823 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573317 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00749489 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586875 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638461 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977653 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779917 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658689 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00538344 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798975 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816861 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822092 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870884 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527339 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545673 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029527 s, 35.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.331001 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601904 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898075 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14725 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977432 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962516 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143065 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12809 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219685 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205305 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23432 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245524 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301598 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33479 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275348 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328743 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302989 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151508 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115293 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976206 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594892 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784651 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783112 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123943 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789058 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847792 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10746 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103123 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116853 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149613 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23232 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158458 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193761 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.262453 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360675 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810838 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365357 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1137 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590899 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131072 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119757 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267509 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063934 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225567 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137893 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2035 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146567 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141606 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157688 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270617 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567296 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025878 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334499 s, 31.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209569 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319738 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732957 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352209 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383944 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717742 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824837 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067157 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403262 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472953 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557494 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083219 s, 12.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096413 s, 10.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (15s) == sanityn test 77e: check TBF NID nrs policy ============ 16:55:04 (1787864104) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.32@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.2512 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363478 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378512 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359523 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666323 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121767 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686135 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603499 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13046 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889945 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138346 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409866 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102557 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208952 s, 5.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217996 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132084 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120635 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127578 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119271 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842269 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865755 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888219 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097648 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912144 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972059 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103516 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104082 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116011 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161128 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110869 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119915 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125595 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184097 s, 5.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.270984 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105572 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235272 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110419 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121764 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144859 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988104 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180077 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166199 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218803 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262474 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251652 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227794 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229191 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219329 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223018 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237994 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226613 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176499 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195779 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600703 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14087 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160407 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1646 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242542 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102582 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138762 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12909 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220413 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764549 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223999 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572315 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197005 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.329608 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406038 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745289 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147707 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681215 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430404 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101917 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578007 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295121 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686993 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169718 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173442 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186277 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17424 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200398 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136874 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220328 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621468 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761106 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645892 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687282 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671624 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786222 s, 13.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107857 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749277 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093122 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856259 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081052 s, 12.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871463 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110499 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110915 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103923 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116364 s, 9.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25233 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313177 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439398 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733137 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494214 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113652 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108336 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235029 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158467 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106542 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725947 s, 14.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110137 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140541 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164325 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256152 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298799 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27842 s, 3.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00332625 s, 315 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00249442 s, 420 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00361579 s, 290 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595907 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611187 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813922 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137844 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00352437 s, 298 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562238 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519061 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587613 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531654 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486947 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470326 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452884 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532179 s, 19.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (21s) == sanityn test 77f: check TBF JobID nrs policy ========== 16:55:25 (1787864125) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.242739 s, 69.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477698 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305979 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757538 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545067 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268963 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138643 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140968 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122694 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571968 s, 18.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17067 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194689 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276234 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267433 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270002 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308039 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.356441 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251236 s, 417 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750152 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591185 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101832 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106212 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684102 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749477 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00640622 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120018 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112084 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578735 s, 18.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481088 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771975 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290477 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365314 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233333 s, 44.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201103 s, 83.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385856 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508865 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107837 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990336 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109869 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115164 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10294 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14906 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130557 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145549 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167437 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197596 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208972 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159731 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22055 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225051 s, 4.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763955 s, 13.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679208 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086353 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908808 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516902 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579695 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729721 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636052 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636371 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102715 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104821 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982669 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124702 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103018 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854025 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957053 s, 11.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.22839 s, 73.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269046 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741252 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963855 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106446 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131136 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11792 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745896 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136803 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147343 s, 7.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175075 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191186 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229758 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23692 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24211 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227861 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314031 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532592 s, 19.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102359 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593122 s, 17.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604789 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622371 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627013 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471983 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063994 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983878 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146982 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125025 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120815 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711835 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109903 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740481 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194803 s, 53.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.449275 s, 37.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286809 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403861 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064951 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465038 s, 22.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860241 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797175 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866576 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455325 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801975 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102405 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114109 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146397 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151687 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074236 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169393 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176894 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759245 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488042 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400359 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047046 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506962 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516374 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764049 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558652 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344748 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385026 s, 27.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669076 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730919 s, 14.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598147 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531484 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734779 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729382 s, 14.4 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (21s) == sanityn test 77g: Change TBF type directly ============ 16:55:46 (1787864146) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 16:55:57 (1787864157) oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 PASS 77h (12s) == sanityn test 77i: Change rank of TBF rule ============= 16:56:09 (1787864169) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (21s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 16:56:31 (1787864191) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427653 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570916 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494796 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211162 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754979 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645061 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157525 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303601 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245678 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307944 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254764 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29766 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370749 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235359 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290822 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375535 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382028 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29345 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242795 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279158 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265549 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252865 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247553 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313132 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278599 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289236 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267931 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293085 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268613 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27048 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270381 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296474 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271514 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=2 removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.10538 s, 11.5 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5985 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (65s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 16:57:35 (1787864255) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31353 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882082 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812204 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674885 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110995 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158559 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692754 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170348 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166411 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160035 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192308 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903127 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227004 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140899 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19136 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153304 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235764 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158742 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446851 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413336 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576875 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672561 s, 15.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682173 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020017 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031486 s, 33.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761856 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492192 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387179 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062252 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822593 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473629 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00708036 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547881 s, 19.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9282 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9798 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25785 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627153 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820121 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086629 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577194 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145395 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152413 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17606 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162427 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179026 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147559 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165234 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149385 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186099 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225671 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183306 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214852 s, 4.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659095 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054336 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774045 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473833 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677651 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900621 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701348 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06711 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511718 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596135 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106358 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953319 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744234 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789054 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816477 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827679 s, 12.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9165 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9321 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (146s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 17:00:02 (1787864402) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29462 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428554 s, 24.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704249 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712114 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467416 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246895 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724345 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641076 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564184 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127261 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145937 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138601 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171005 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173934 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150757 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194019 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23924 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643325 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662948 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067708 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715353 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748403 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076249 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777129 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797907 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793431 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819337 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825001 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797353 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809862 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855186 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910321 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097957 s, 10.7 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9499 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.973 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.27629 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398372 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406224 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665349 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490742 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380849 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777257 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102564 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107146 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731252 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706551 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147862 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177773 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147523 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16136 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188302 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199012 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521966 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566468 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605687 s, 17.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732727 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560193 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807174 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585179 s, 17.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740506 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580468 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653038 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069084 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673989 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760369 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10064 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723554 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880764 s, 11.9 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0078 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9326 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (151s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 17:02:33 (1787864553) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.602724 s, 27.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342739 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398952 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229197 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544892 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109752 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361295 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692971 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306264 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220185 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220007 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199105 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120991 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211233 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236277 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202718 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288016 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348949 s, 30.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271326 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497991 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238696 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198214 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044732 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267616 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595504 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561331 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395035 s, 26.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661482 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519968 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159289 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058078 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606265 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650329 s, 16.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.21658 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91346 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.6796 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364894 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168631 s, 62.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356437 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502273 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925204 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171138 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160697 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143358 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104375 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175167 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131729 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180729 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231415 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212596 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206543 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303773 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9de003eca000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158828 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102645 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117271 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150787 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117992 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120783 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129093 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140154 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143101 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147126 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153482 s, 6.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de003eca000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156864 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155963 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220517 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172763 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167471 s, 6.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.37199 s, 14.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90773 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.484024 s, 34.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308489 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339132 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475249 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300956 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129647 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169375 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164754 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222565 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186756 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191483 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257535 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274928 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278233 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286215 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273417 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270735 s, 3.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490663 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683673 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627706 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723984 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429167 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788084 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280252 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616852 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448165 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633337 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547316 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693753 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749269 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121311 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196833 s, 53.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125142 s, 8.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.94936 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.32579 s, 19.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.598101 s, 28.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050975 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185838 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643644 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849902 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776852 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940086 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108666 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194418 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240127 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241679 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227212 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228727 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257101 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248328 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167333 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190927 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457852 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541515 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574321 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103311 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628484 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712775 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650896 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717383 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766904 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778818 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829654 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744035 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838208 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835413 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865587 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101122 s, 10.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1162 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91385 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.02363 s, 14.9 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.8863 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.99353 s, 16.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319761 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058696 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064506 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954293 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826587 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124332 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098159 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13999 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125685 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124583 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229138 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21386 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218599 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224374 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286341 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32541 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725072 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766569 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612908 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656772 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122126 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426268 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685283 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990201 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651684 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109657 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954284 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112645 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699418 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947973 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093658 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120082 s, 8.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8467 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6361 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.347308 s, 48.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0616194 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247221 s, 42.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497004 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709901 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117168 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117371 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102142 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111645 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144008 s, 7.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143941 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108302 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107623 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159501 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164724 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167312 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170626 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994211 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311934 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399411 s, 26.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108727 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836082 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10684 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799979 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10885 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402353 s, 26.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727231 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510873 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540287 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902505 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891025 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532389 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581917 s, 18.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.83093 s, 13.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98553 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.628573 s, 26.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594999 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806479 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153388 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115807 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124992 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26857 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237987 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235895 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283054 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195486 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25538 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384656 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384195 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404084 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320212 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.396703 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123231 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619446 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695068 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122131 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0975902 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768913 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645925 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832889 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753875 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498704 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928351 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938641 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705272 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725136 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116909 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127365 s, 8.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.7029 s, 9.0 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9506 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.4788 s, 10.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90458 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (440s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 17:09:53 (1787864993) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.611704 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611415 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686666 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462441 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197261 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619795 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107901 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721664 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108588 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556108 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643871 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160771 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13476 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187109 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196232 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204019 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207137 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596816 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654898 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494605 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0578575 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555939 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606436 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721239 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055202 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590734 s, 17.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864051 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488667 s, 21.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688446 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681163 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629921 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556204 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765391 s, 13.7 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.68351 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91322 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (50s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 17:10:43 (1787865043) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.428319 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394513 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510713 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918877 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935772 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121993 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121578 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138874 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113922 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129317 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166627 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142891 s, 7.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146819 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160243 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13739 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186853 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149727 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761332 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432243 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692148 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042458 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498875 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485098 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848925 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823175 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041342 s, 25.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213431 s, 49.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538931 s, 19.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438092 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538403 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534936 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00357961 s, 293 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225105 s, 46.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.70738 s, 15.6 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90455 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (47s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 17:11:30 (1787865090) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.558601 s, 30.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373729 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102558 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112465 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184061 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667526 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135511 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154132 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121391 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145062 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102949 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213787 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208624 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166723 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237774 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232749 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242162 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385118 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422411 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290206 s, 36.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00350875 s, 299 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00435731 s, 241 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581635 s, 180 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727036 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101415 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074251 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803622 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775466 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142514 s, 73.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130972 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00950702 s, 110 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853783 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862667 s, 12.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.49857 s, 12.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.11174 s, 20.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (45s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 17:12:15 (1787865135) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.587824 s, 28.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510642 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046576 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023377 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563993 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655913 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825764 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170644 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146478 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119362 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775038 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863915 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833806 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266509 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181588 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325888 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303794 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610791 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530285 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309445 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119189 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053021 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120894 s, 86.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126175 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00828159 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00227444 s, 461 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106145 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0557715 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593267 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028282 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694476 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221231 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343317 s, 30.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.6119 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88824 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.10221 s, 17.2 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87394 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (87s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 17:13:42 (1787865222) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00489 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489417 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985823 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678711 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113568 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123594 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169143 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141906 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184172 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150322 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163619 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155794 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202025 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195944 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19283 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176225 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21095 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206895 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299165 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485887 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463523 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295374 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572834 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340009 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384729 s, 27.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574722 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654335 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509235 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518462 s, 20.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691943 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528292 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698814 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583762 s, 18.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8446 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9546 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (72s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 17:14:54 (1787865294) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.419818 s, 40.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481612 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419582 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113947 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134027 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138697 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155606 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151308 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120852 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140387 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199034 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18432 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158878 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187962 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248819 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250034 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277909 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099819 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108098 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920272 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14246 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120287 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111973 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132458 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115378 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126058 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953481 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14289 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16899 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141781 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156961 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153112 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19418 s, 5.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.91631 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94405 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.609221 s, 27.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044748 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594575 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203318 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202552 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17089 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201284 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221354 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264229 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199221 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216203 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287098 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241506 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248359 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275202 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230106 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332774 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598086 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116521 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770396 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108904 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985981 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130392 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13519 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684482 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935207 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793586 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.080607 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084171 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810092 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153533 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834409 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107566 s, 9.7 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.396 s, 10.1 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90475 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.8681 s, 9.6 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94194 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (139s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 17:17:13 (1787865433) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:) Stopping client oleg432-client.virtnet /mnt/lustre opts: Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg432-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg432-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg432-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg432-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/68879/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.32,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg432-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9de0142cf800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9de02eb95800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de0142cf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de02eb95800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.396408 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472304 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358396 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067209 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131673 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228609 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236158 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214633 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25156 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250329 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277482 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286133 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245182 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263063 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248482 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28302 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342379 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263076 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200679 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032011 s, 32.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591864 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396005 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326868 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809109 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032019 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752431 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429423 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425229 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512206 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404235 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593316 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645335 s, 16.2 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.59024 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.20723 s, 20.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.765931 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642353 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855272 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112135 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194678 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859327 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209886 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135065 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276946 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312453 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184256 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412782 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242529 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360527 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de0142cf800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285614 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320601 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375421 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182053 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151376 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199425 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112687 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15127 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145821 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133212 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222823 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230866 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149703 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236013 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170321 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170632 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274151 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18945 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191236 s, 5.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9de0142cf800.lock_unused_count=7 removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1655 s, 9.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0783 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (209s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 17:20:42 (1787865642) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (17s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 17:20:59 (1787865659) On MGS 192.168.204.132, active = nodemap.active=1 On MGS 192.168.204.132, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.626594 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526578 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044152 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115455 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139193 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139498 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160232 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165267 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159258 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155017 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160834 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179871 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188948 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200525 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189665 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161666 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183759 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642527 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364485 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542105 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474604 s, 22.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387699 s, 27.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487068 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527019 s, 19.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565082 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505589 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055562 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595863 s, 17.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919775 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062132 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622606 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786317 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672831 s, 15.6 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.89754 s, 15.2 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.13722 s, 20.4 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.204.132, TBF.id = On MGS 192.168.204.132, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.554997 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753772 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855725 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596278 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137096 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976111 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127411 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114598 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14939 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119852 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144633 s, 7.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de0142cf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172598 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201793 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248515 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193696 s, 5.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9de0142cf800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309167 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277374 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914192 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986306 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935848 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127415 s, 82.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976326 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270135 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254221 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258184 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105956 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673476 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325104 s, 32.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371716 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351715 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401032 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379992 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501341 s, 20.9 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.47591 s, 12.4 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.04346 s, 34.5 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg432-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 On MGS 192.168.204.132, active = nodemap.active=1 On MGS 192.168.204.132, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.507549 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451606 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715572 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752168 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874474 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150484 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130299 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217789 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208039 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197977 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209326 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247796 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315955 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280783 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329005 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289733 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310977 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349312 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411907 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625088 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430805 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540261 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502659 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509695 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950651 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052397 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586818 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063812 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686287 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07038 s, 14.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753767 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752435 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856178 s, 12.2 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.73294 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87998 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.204.132, TBF.id = On MGS 192.168.204.132, active = nodemap.active=0 PASS 77kj (163s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 17:23:42 (1787865822) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (9s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 17:23:51 (1787865831) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg432-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 4.12513 s, 254 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg432-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0736 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 17:24:49 (1787865889) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.458652 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870724 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733356 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104685 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12104 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102397 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148011 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192807 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103931 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182849 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117507 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162616 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133106 s, 7.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de0142cf800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287297 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301377 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179958 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264725 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351989 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458797 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309127 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101021 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115038 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472248 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724307 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07466 s, 14.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902358 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537817 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527236 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125633 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00871616 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556551 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610168 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623265 s, 16.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.18871 s, 12.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16465 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.660247 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537203 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673419 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141279 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204463 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19489 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224453 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236862 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266989 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266914 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258209 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281457 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293108 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292825 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322168 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305321 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324315 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9de0142cf800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9de0142cf800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393991 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398328 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370642 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386776 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.386094 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40147 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382061 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383907 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425768 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393315 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410326 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438731 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394193 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421343 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424131 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427494 s, 2.5 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg432-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg432-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.68978 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.15019 s, 20.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 17:26:18 (1787865978) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (11s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 17:26:29 (1787865989) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (122s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 17:28:31 (1787866111) mds.MDS.mdt.nrs_policies=tbf oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg432-client: oleg432-server: ssh exited with exit code 36 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 oleg432-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg432-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg432-client: oleg432-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (41s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 17:29:12 (1787866152) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273292 kill md_thread xargs PASS 77r (58s) == sanityn test 77s: Check TBF LRU shrinker ============== 17:30:10 (1787866210) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 8 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 8 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (70s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 17:31:20 (1787866280) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 17:31:31 (1787866291) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935627 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:31:33.000000000 -0400 Modify: 2026-08-27 17:31:33.000000000 -0400 Change: 2026-08-27 17:31:33.000000000 -0400 Birth: 2026-08-27 17:31:33.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 17:31:50 (1787866310) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 17:31:55 (1787866315) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 17:31:59 (1787866319) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 17:32:03 (1787866323) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 17:32:06 (1787866326) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 17:32:10 (1787866330) - open/close 649 (time 1787866345.74 total 10.00 last 64.87) - open/close 1328 (time 1787866355.75 total 20.02 last 67.81) - open/close 2016 (time 1787866365.76 total 30.03 last 68.72) total: 2500 open/close in 37.81 seconds: 66.11 ops/second snapshot_time 1787866471.669743023 secs.nsecs start_time 1787866333.479193736 secs.nsecs elapsed_time 138.190549287 secs.nsecs open 2505 samples [usecs] 217 18644 4559890 13212592816 close 2505 samples [usecs] 220 6171 1103659 945063881 mknod 2501 samples [usecs] 654 18014 3968196 10237334720 unlink 2500 samples [usecs] 871 127284 8193732 141546135768 mkdir 2 samples [usecs] 5856 11549 17405 167672137 rmdir 2 samples [usecs] 10032 16741 26773 380902105 rename 2500 samples [usecs] 565 43112 19233519 282041979445 getattr 3763 samples [usecs] 10 8792 318274 541281574 parallel_rename_file 2500 samples [usecs] 565 43112 19233519 282041979445 crossdir_rename 2500 samples [usecs] 565 43112 19233519 282041979445 readpage 34 samples [usecs] 7058 91132 1108573 51634111211 PASS 81d (148s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 17:34:39 (1787866479) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7639 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7639 PASS 82 (9s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 17:34:47 (1787866487) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 17:34:51 (1787866491) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 17:35:04 (1787866504) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1787866506 Finishing test test1 at 1787866510 PASS 85 (11s) == sanityn test 90: open/create and unlink striped directory ========================================================== 17:35:15 (1787866515) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 17:35:19 (1787866519) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 17:35:23 (1787866523) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 17:35:26 (1787866526) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5749 0x1675 0x240000400 1 5711 0x164f 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5750 0x1676 0x240000400 1 5712 0x1650 0x280000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 17:35:44 (1787866544) osc.lustre-OST0000-osc-ffff9de0142cf800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9de02eb95800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9de0142cf800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9de02eb95800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.215805 s, 38.0 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.97999 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290137 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0762522 s, 53.7 kB/s osc.lustre-OST0000-osc-ffff9de0142cf800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9de02eb95800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de0142cf800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9de02eb95800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 17:35:58 (1787866558) llite.lustre-ffff9de0142cf800.fast_read=0 llite.lustre-ffff9de02eb95800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0461138 s, 45.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9de0142cf800.fast_read=1 llite.lustre-ffff9de02eb95800.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 17:36:17 (1787866577) llite.lustre-ffff9de0142cf800.fast_read=0 llite.lustre-ffff9de02eb95800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00686834 s, 1.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9de0142cf800.fast_read=1 llite.lustre-ffff9de02eb95800.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 17:36:31 (1787866591) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (5s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 17:36:36 (1787866596) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00931327 s, 440 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (10s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 17:36:46 (1787866606) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191132 s, 11.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (9s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 17:36:56 (1787866616) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170782 s, 12.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 17:37:03 (1787866623) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00813741 s, 1.5 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 17:37:11 (1787866631) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253563 s, 162 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 17:37:19 (1787866639) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0131319 s, 312 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 17:37:27 (1787866647) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105464 s, 388 kB/s PASS 101c (8s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 17:37:35 (1787866655) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 2c 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf2c:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (8s) == sanityn test 103: Test size correctness with lockahead ========================================================== 17:37:43 (1787866663) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1787866665 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1787866668 Finishing test test23 at 1787866669 PASS 103 (13s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 17:37:56 (1787866676) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 17:37:59 (1787866679) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 17:38:22 (1787866702) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 17:38:26 (1787866706) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00650361 s, 157 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938163 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:38:27.000000000 -0400 Modify: 2026-08-27 17:38:27.000000000 -0400 Change: 2026-08-27 17:38:27.000000000 -0400 Birth: 2026-08-27 17:38:28.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938163 -rw-r--r-- 1 0 0 1787866708 1787866707 1787866707 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938163 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:38:27.000000000 -0400 Modify: 2026-08-27 17:38:27.000000000 -0400 Change: 2026-08-27 17:38:27.000000000 -0400 Birth: 2026-08-27 17:38:28.000000000 -0400 1787866707 1024 1 PASS 106b (7s) == sanityn test 106c: Verify statx attributes mask ======= 17:38:33 (1787866713) PASS 106c (7s) == sanityn test 107a: Basic grouplock conflict =========== 17:38:40 (1787866720) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.285126 s, 36.8 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5759 0x167f 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 17:38:49 (1787866729) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.336564 s, 31.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5719 0x1657 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7639 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (13s) == sanityn test 108a: lseek: parallel updates ============ 17:39:03 (1787866743) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00648709 s, 1.5 MB/s llite.lustre-ffff9de0142cf800.hybrid_io=0 llite.lustre-ffff9de02eb95800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.68496 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 17:39:15 (1787866755) 192.168.204.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre 192.168.204.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.132@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg432-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 PASS 109 (65s) == sanityn test 110: do not grant another lock on resend ========================================================== 17:40:20 (1787866820) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 17:40:22.000000000 -0400 Modify: 2026-08-27 17:40:22.000000000 -0400 Change: 2026-08-27 17:40:22.000000000 -0400 Birth: 2026-08-27 17:40:22.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg432-client.virtnet oleg432-client.virtnet PASS 110 (91s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 17:41:51 (1787866911) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 17:41:55 (1787866915) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 17:41:59 (1787866919) lustre-OST0000-osc-ffff9de00329e800 active. lustre-OST0001-osc-ffff9de00329e800 active. lustre-OST0000-osc-ffff9de00329e800 active. lustre-OST0001-osc-ffff9de00329e800 active. MGC192.168.204.132@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 17:42:08 (1787866928) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 17:42:14 (1787866934) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 17:42:18 (1787866938) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 17:42:23 (1787866943) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00760411 s, 2.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0428015 s, 478 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 17:42:33 (1787866953) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1011 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1001 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1719 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1568 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1154 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1299 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1821 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1535 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2428 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.223 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2381 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2905 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2861 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3458 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.397 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3499 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4476 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3957 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9de00329e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9de00329e800.ost_server_uuid in FULL state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9899 sec ============== 17:43:57 (1787867037) === sanityn: start cleanup 17:43:58 (1787867038) === Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg432-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 17:47:54 (1787867274) ===