-----============= acceptance-small: sanityn ============----- Fri May 8 18:12:11 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg457-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 18:12:29 (1778278349) === Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Mount client oleg457-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg457-client.virtnet: executing check_config_client /mnt/lustre oleg457-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg457-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff934cc5473800.idle_timeout=debug osc.lustre-OST0000-osc-ffff934cd1bbc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cc5473800.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cd1bbc800.idle_timeout=debug disable quota as required oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 18:13:01 (1778278381) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251502 s, 41.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7568] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 18:13:03 (1778278383) PASS 0 (7s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 18:13:10 (1778278390) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 18:13:15 (1778278395) -rw-r--r-- 1 root root 0 May 8 18:13 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (5s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 18:13:20 (1778278400) -rw-r--r-- 1 root root 0 May 8 18:13 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (5s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 18:13:25 (1778278405) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (5s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 18:13:30 (1778278410) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 18:13:36 (1778278416) total 1 -rwxrwxrwx 1 root root 0 May 8 18:13 f2a -rwxrwxrwx 1 root root 0 May 8 18:13 f2b total 1 -rwxrwxrwx 1 root root 0 May 8 18:13 f2a -rwxrwxrwx 1 root root 0 May 8 18:13 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0189628 s, 27.0 kB/s PASS 2e (5s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 18:13:41 (1778278421) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 18:13:43 (1778278423) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.374821 s, 5.6 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 18:13:50 (1778278430) PASS 3 (5s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 18:13:55 (1778278435) PASS 4 (7s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 18:14:02 (1778278442) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 18:14:07 (1778278447) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 18:14:12 (1778278452) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 18:14:17 (1778278457) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (7s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 18:14:24 (1778278464) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 18:14:32 (1778278472) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533009 s, 19.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0322887 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 18:14:38 (1778278478) 1+0 records in 1+0 records out 1 byte copied, 0.0196335 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103437 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0206588 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.01003 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00965574 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00838143 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0203897 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0119004 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128001 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00969698 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00813654 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.018698 s, 0.1 kB/s PASS 10a (6s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 18:14:45 (1778278485) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0265983 s, 115 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0346212 s, 118 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000270876 s, 11.3 MB/s PASS 10b (5s) == sanityn test 11: execution of file opened for write should return error ============================================================== 18:14:50 (1778278490) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 18:14:57 (1778278497) warning: '-runtime' deprecated, use '-t runtime' instead - link 391 (time 1778278509.22 total 10.02 last 39.01) - link 747 (time 1778278519.26 total 20.06 last 35.47) - link 1137 (time 1778278529.27 total 30.07 last 38.97) - link 1652 (time 1778278539.27 total 40.07 last 51.46) - link 2193 (time 1778278549.28 total 50.07 last 54.10) - link 2677 (time 1778278559.28 total 60.08 last 48.35) - link 3227 (time 1778278569.29 total 70.09 last 54.99) - link 3777 (time 1778278579.30 total 80.10 last 54.94) - link 4256 (time 1778278589.30 total 90.10 last 47.88) total: 4712 link in 100.05 seconds: 47.10 ops/second using seed 1542533882 running for 100 seconds - stat 5495 (time 1778278511 ; total 11 ; last 11) - stat 10000 (time 1778278519 ; total 19 ; last 8) - stat 15517 (time 1778278530 ; total 30 ; last 11) - stat 20000 (time 1778278536 ; total 36 ; last 6) - stat 27432 (time 1778278547 ; total 47 ; last 11) - stat 30000 (time 1778278551 ; total 51 ; last 4) - stat 33223 (time 1778278562 ; total 62 ; last 11) - stat 36865 (time 1778278573 ; total 73 ; last 11) - stat 40000 (time 1778278582 ; total 82 ; last 9) - stat 43040 (time 1778278593 ; total 93 ; last 11) total: 45016 stats in 100 seconds: 450.160004 stats/second - unlinked 0 (time 1778278550 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile4712) error: No such file or directory total: 4711 unlinks in 97 seconds: 48.567009 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21360) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21362) - No such process PASS 12 (153s) == sanityn test 13: test directory page revocation ======= 18:17:30 (1778278650) f13 f13 PASS 13 (6s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 18:17:36 (1778278656) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 22546 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 18:17:39 (1778278659) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 23125 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 18:17:42 (1778278662) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23703 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 18:17:45 (1778278665) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24284 Terminated $DIR1/$tdir/sleep 60 PASS 14c (3s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 18:17:48 (1778278668) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24865 Terminated $DIR1/$tdir/sleep 60 PASS 14d (4s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 18:17:52 (1778278672) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (35s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 18:18:27 (1778278707) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.256205 s, 32.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.216763 s, 38.7 MB/s Chance of close/open is 1 in 50 Seed set to 8709 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1778278710.001880 mapwrite 0x7977e2 thru 0x7a03de (0x8bfd bytes) 000003[1] 1778278710.099271 mapwrite 0x80d895 thru 0x81156d (0x3cd9 bytes) 000005[1] 1778278710.218705 mapread 0x240d37 thru 0x2483e6 (0x76b0 bytes) 000008[1] 1778278710.254310 read 0x6c3c87 thru 0x6caab7 (0x6e31 bytes) 000012[1] 1778278710.293784 mapwrite 0x13ce8d thru 0x13d8e4 (0xa58 bytes) 000013[1] 1778278710.346088 mapwrite 0x573508 thru 0x574fd6 (0x1acf bytes) 000014[1] 1778278710.380157 mapwrite 0x3295b2 thru 0x333d26 (0xa775 bytes) 000015[1] 1778278710.438649 write 0x5c5861 thru 0x5c85e6 (0x2d86 bytes) 000016[0] 1778278710.456528 write 0x338aa2 thru 0x346873 (0xddd2 bytes) 000020[1] 1778278710.553737 read 0x722e8b thru 0x725a86 (0x2bfc bytes) 000023[1] 1778278710.602119 write 0x37774d thru 0x38310e (0xb9c2 bytes) 000025[0] 1778278710.635593 read 0x318994 thru 0x31df25 (0x5592 bytes) truncating to largest ever: 0x15a74c 000028[1] 1778278710.680740 trunc from 0x81156e to 0x15a74b (0x6b6e22 bytes) 000032[0] 1778278710.739762 write 0x677f98 thru 0x67c84e (0x48b7 bytes) 000035[0] 1778278710.795812 write 0x547b77 thru 0x553dc9 (0xc253 bytes) 000036[0] 1778278710.818255 mapread 0x2c6460 thru 0x2cd73f (0x72e0 bytes) 000037[1] 1778278710.878539 mapread 0x25da46 thru 0x266f68 (0x9523 bytes) truncating to largest ever: 0x978445 000040[1] 1778278711.055703 trunc from 0x67c84f to 0x978444 (0x2fbbf6 bytes) 000041[0] 1778278711.127351 read 0x7a392a thru 0x7abd22 (0x83f9 bytes) 000042[0] 1778278711.162507 mapread 0x0129d9 thru 0x018178 (0x57a0 bytes) 000044[1] 1778278711.228152 trunc from 0x978445 to 0x43f1fc (0x539248 bytes) 000049[0] 1778278711.335202 mapread 0x277d56 thru 0x280c89 (0x8f34 bytes) 000052[0] 1778278711.413067 mapwrite 0x1d847b thru 0x1e532d (0xceb3 bytes) 000053[1] 1778278711.498296 write 0x51ab3a thru 0x5255b1 (0xaa78 bytes) 000057[0] 1778278711.578900 write 0x602c1a thru 0x609d06 (0x70ed bytes) 000060[1] 1778278711.640960 read 0x475b2d thru 0x47ce22 (0x72f6 bytes) 000061[1] 1778278711.670108 mapread 0x082144 thru 0x08e59f (0xc45c bytes) 000063[1] 1778278711.720435 write 0x56f531 thru 0x57328a (0x3d5a bytes) 000066[0] 1778278711.789395 write 0x935b31 thru 0x93e16c (0x863c bytes) 000067[0] 1778278711.823772 read 0x8d4d68 thru 0x8e1af2 (0xcd8b bytes) 000068[0] 1778278711.862314 trunc from 0x93e16d to 0x67a460 (0x2c3d0c bytes) 000069[0] 1778278711.911577 write 0x31a234 thru 0x31ec88 (0x4a55 bytes) 000070[0] 1778278711.923899 mapwrite 0x8f9658 thru 0x900786 (0x712f bytes) 000072[0] 1778278712.039331 mapwrite 0x92a4c7 thru 0x9321be (0x7cf8 bytes) 000079[0] 1778278712.176538 read 0x624a23 thru 0x628894 (0x3e72 bytes) 000080[0] 1778278712.182743 write 0x167bcf thru 0x176d76 (0xf1a8 bytes) 000086[0] 1778278712.256194 write 0x1f9fc9 thru 0x207c94 (0xdccc bytes) 000087[0] 1778278712.288558 read 0x7db648 thru 0x7eaa72 (0xf42b bytes) 000089[1] 1778278712.331299 write 0x91ccab thru 0x9246ec (0x7a42 bytes) 000091[0] 1778278712.382040 trunc from 0x9321bf to 0x07f19f (0x8b301f bytes) 000092[0] 1778278712.434088 mapwrite 0x12d65e thru 0x137605 (0x9fa8 bytes) 000095[0] 1778278712.557745 mapwrite 0x24e10e thru 0x256b0d (0x8a00 bytes) 000097[0] 1778278712.665686 trunc from 0x256b0e to 0x768719 (0x511c0c bytes) 000098[0] 1778278712.685073 trunc from 0x76871a to 0x8ecd03 (0x1845ea bytes) 000099[1] 1778278712.715678 read 0x234cb2 thru 0x23cc05 (0x7f54 bytes) 000104[1] 1778278712.817396 read 0x1e311a thru 0x1f1960 (0xe847 bytes) 000105[1] 1778278712.841015 write 0x21126f thru 0x2196e5 (0x8477 bytes) 000110[0] 1778278712.971690 write 0x13c225 thru 0x1434f7 (0x72d3 bytes) 000111[0] 1778278713.005275 mapread 0x51019d thru 0x51035f (0x1c3 bytes) 000115[1] 1778278713.013506 read 0x04d03b thru 0x0523c4 (0x538a bytes) 000116[0] 1778278713.071408 mapwrite 0x5f6c2a thru 0x5f8627 (0x19fe bytes) 000117[0] 1778278713.148838 write 0x720204 thru 0x7297f8 (0x95f5 bytes) 000119[1] 1778278713.177543 mapwrite 0x78f34c thru 0x79f0cd (0xfd82 bytes) 000120[0] 1778278713.263668 write 0x6ced47 thru 0x6cf097 (0x351 bytes) 000123[0] 1778278713.306724 write 0x0752e1 thru 0x07540b (0x12b bytes) 000126[1] 1778278713.377297 mapread 0x1a8947 thru 0x1a9919 (0xfd3 bytes) 000127[1] 1778278713.418675 mapwrite 0x4ff69f thru 0x500ab1 (0x1413 bytes) 000129[1] 1778278713.495151 write 0x7198ad thru 0x71cde2 (0x3536 bytes) 000130[0] 1778278713.503829 write 0x23ff28 thru 0x249f82 (0xa05b bytes) 000131[1] 1778278713.538150 mapwrite 0x140d54 thru 0x14a1e1 (0x948e bytes) 000132[0] 1778278713.637817 mapread 0x8339dd thru 0x8392c8 (0x58ec bytes) 000138[1] 1778278713.751287 read 0x06b65b thru 0x06e5b3 (0x2f59 bytes) 000140[0] 1778278713.785060 trunc from 0x8ecd04 to 0x61853c (0x2d47c7 bytes) 000142[0] 1778278713.851158 write 0x045048 thru 0x04d9ea (0x89a3 bytes) 000146[0] 1778278713.907259 mapread 0x1befa4 thru 0x1befcb (0x28 bytes) 000151[0] 1778278713.978548 trunc from 0x61853d to 0x59835b (0x801e1 bytes) 000152[1] 1778278714.016048 mapwrite 0x4d1420 thru 0x4d3d0b (0x28ec bytes) 000153[1] 1778278714.117102 trunc from 0x59835c to 0x219cc2 (0x37e699 bytes) 000154[1] 1778278714.159724 write 0x0667dc thru 0x0698e3 (0x3108 bytes) 000160[1] 1778278714.252434 trunc from 0x219cc3 to 0x955dd5 (0x73c113 bytes) 000161[1] 1778278714.295774 mapwrite 0x164925 thru 0x17027f (0xb95b bytes) 000163[0] 1778278714.379825 mapwrite 0x4ddb4a thru 0x4e209a (0x4551 bytes) 000165[0] 1778278714.479873 mapwrite 0x0c4ba2 thru 0x0cc9d0 (0x7e2f bytes) 000167[1] 1778278714.554319 mapwrite 0x0cd42e thru 0x0d2831 (0x5404 bytes) 000168[1] 1778278714.626478 write 0x25bcd9 thru 0x260bf3 (0x4f1b bytes) 000169[0] 1778278714.641316 write 0x95758e thru 0x960fe5 (0x9a58 bytes) 000170[1] 1778278714.657363 mapread 0x012775 thru 0x01a261 (0x7aed bytes) 000172[1] 1778278714.718368 read 0x0bdf3d thru 0x0c77e4 (0x98a8 bytes) 000173[1] 1778278714.740058 mapwrite 0x7b862e thru 0x7c665f (0xe032 bytes) 000175[1] 1778278714.842780 write 0x5d6429 thru 0x5dbd19 (0x58f1 bytes) 000176[0] 1778278714.851207 trunc from 0x960fe6 to 0x39280d (0x5ce7d8 bytes) 000177[1] 1778278714.899454 mapwrite 0x4ccc8e thru 0x4dcc01 (0xff74 bytes) 000179[0] 1778278714.993324 write 0x429e35 thru 0x43289c (0x8a68 bytes) 000180[1] 1778278715.020818 mapwrite 0x2d9c44 thru 0x2e591e (0xbcdb bytes) 000181[0] 1778278715.113070 mapread 0x262fd4 thru 0x264769 (0x1796 bytes) 000184[0] 1778278715.197446 mapread 0x150e87 thru 0x15abbf (0x9d39 bytes) 000187[1] 1778278715.229795 mapwrite 0x1c36b4 thru 0x1d3583 (0xfed0 bytes) 000193[1] 1778278715.425269 trunc from 0x4dcc02 to 0x223cb3 (0x2b8f4e bytes) 000195[1] 1778278715.468070 write 0x09684a thru 0x0a2732 (0xbee9 bytes) 000198[0] 1778278715.503018 trunc from 0x223cb4 to 0x2fb5f3 (0xd7940 bytes) 000199[0] 1778278715.552374 write 0x4e3b84 thru 0x4ecb82 (0x8fff bytes) 000200[0] 1778278715.555680 trunc from 0x4ecb83 to 0x2718a4 (0x27b2de bytes) 000202[1] 1778278715.601091 read 0x1d58c3 thru 0x1db380 (0x5abe bytes) 000203[0] 1778278715.629868 mapwrite 0x8047cc thru 0x806e13 (0x2648 bytes) 000206[1] 1778278715.777731 mapwrite 0x9efbd8 thru 0x9f24a2 (0x28cb bytes) 000207[1] 1778278715.876374 write 0x9edcc2 thru 0x9f21bd (0x44fc bytes) 000209[0] 1778278715.926371 trunc from 0x9f24a3 to 0x5a1ac3 (0x4509df bytes) 000211[1] 1778278716.007221 write 0x577988 thru 0x57ca6c (0x50e5 bytes) 000212[0] 1778278716.038590 write 0x2559f9 thru 0x25c502 (0x6b0a bytes) 000217[0] 1778278716.113806 mapwrite 0x9e4e73 thru 0x9ee74f (0x98dd bytes) 000219[1] 1778278716.249344 write 0x998604 thru 0x99e470 (0x5e6d bytes) 000220[1] 1778278716.278642 read 0x9bf7dd thru 0x9cb2da (0xbafe bytes) 000221[1] 1778278716.303548 mapwrite 0x506e85 thru 0x50db66 (0x6ce2 bytes) 000224[1] 1778278716.436277 trunc from 0x9ee750 to 0x37b5a3 (0x6731ac bytes) truncating to largest ever: 0x9b3e53 000225[0] 1778278716.475475 trunc from 0x37b5a4 to 0x9b3e52 (0x6388af bytes) 000227[1] 1778278716.571932 read 0x75f0e3 thru 0x76b179 (0xc097 bytes) 000229[0] 1778278716.615859 write 0x424db8 thru 0x4323bf (0xd608 bytes) 000230[0] 1778278716.641054 mapread 0x0d91ed thru 0x0dedb1 (0x5bc5 bytes) 000231[1] 1778278716.671457 read 0x51e5a0 thru 0x52b791 (0xd1f2 bytes) 000233[1] 1778278716.738004 mapwrite 0x3f4806 thru 0x3fb9e5 (0x71e0 bytes) 000237[1] 1778278716.863857 mapwrite 0x6f40af thru 0x6f4ce3 (0xc35 bytes) 000239[0] 1778278716.951517 mapwrite 0x31e676 thru 0x32c3b2 (0xdd3d bytes) 000240[0] 1778278717.086477 read 0x2a5fe6 thru 0x2b49db (0xe9f6 bytes) 000241[1] 1778278717.108920 mapread 0x92c65e thru 0x92f274 (0x2c17 bytes) 000243[0] 1778278717.162070 write 0x9d2cd9 thru 0x9d782b (0x4b53 bytes) 000245[0] 1778278717.209984 mapwrite 0x50beb7 thru 0x518f4e (0xd098 bytes) 000246[1] 1778278717.307977 mapwrite 0x497ccc thru 0x49d35a (0x568f bytes) 000248[1] 1778278717.441538 mapwrite 0x63b3a8 thru 0x64927f (0xded8 bytes) 000250[0] 1778278717.567589 mapwrite 0x92ed4c thru 0x937398 (0x864d bytes) 000251[1] 1778278717.663938 mapwrite 0x85d456 thru 0x868977 (0xb522 bytes) 000252[0] 1778278717.771580 trunc from 0x9d782c to 0x5b8841 (0x41efea bytes) 000253[0] 1778278717.818576 mapwrite 0x360c1c thru 0x361e30 (0x1215 bytes) truncating to largest ever: 0x9babdb 000254[1] 1778278717.872570 trunc from 0x5b8842 to 0x9babda (0x402399 bytes) 000258[1] 1778278717.970036 mapwrite 0x20daec thru 0x2150fe (0x7613 bytes) 000259[1] 1778278718.036110 mapread 0x2368b7 thru 0x236e6c (0x5b6 bytes) 000261[0] 1778278718.086698 trunc from 0x9babdb to 0x464124 (0x556ab6 bytes) 000262[1] 1778278718.150557 mapread 0x222fbf thru 0x2281d5 (0x5217 bytes) 000264[0] 1778278718.213144 mapread 0x45732b thru 0x464124 (0xcdfa bytes) 000266[0] 1778278718.248854 read 0x0f3d3f thru 0x0ffc8d (0xbf4f bytes) 000269[1] 1778278718.295033 mapwrite 0x2b928d thru 0x2bc4c5 (0x3239 bytes) 000270[1] 1778278718.373224 trunc from 0x464125 to 0x2e2da2 (0x181382 bytes) 000272[1] 1778278718.426959 read 0x077ea3 thru 0x08003c (0x819a bytes) 000273[1] 1778278718.475012 read 0x1910f5 thru 0x198359 (0x7265 bytes) 000274[1] 1778278718.493843 read 0x1743d7 thru 0x175aff (0x1729 bytes) 000275[1] 1778278718.512750 mapread 0x26d1a6 thru 0x278137 (0xaf92 bytes) 000276[0] 1778278718.528970 read 0x05d809 thru 0x06c065 (0xe85d bytes) 000279[0] 1778278718.613144 mapread 0x2961d1 thru 0x2a5cc0 (0xfaf0 bytes) 000282[0] 1778278718.681439 mapread 0x0110cd thru 0x01d4c1 (0xc3f5 bytes) 000284[1] 1778278718.708362 write 0x9df44b thru 0x9e1025 (0x1bdb bytes) 000285[1] 1778278718.742324 read 0x98ba0d thru 0x991da7 (0x639b bytes) 000288[0] 1778278718.767720 read 0x8ba524 thru 0x8c820d (0xdcea bytes) 000292[0] 1778278718.831407 write 0x256bb7 thru 0x259f56 (0x33a0 bytes) 000293[0] 1778278718.882221 write 0x92e45b thru 0x93c40f (0xdfb5 bytes) 000296[1] 1778278718.940054 trunc from 0x9e1026 to 0x19dd4b (0x8432da bytes) 000298[0] 1778278719.003382 mapwrite 0x76cc86 thru 0x777cd0 (0xb04b bytes) 000299[1] 1778278719.126306 write 0x764976 thru 0x7713f0 (0xca7b bytes) 000302[1] 1778278719.212945 write 0x5eb687 thru 0x5f48f8 (0x9272 bytes) 000303[1] 1778278719.254737 mapread 0x16f8ce thru 0x17f15a (0xf88d bytes) 000305[0] 1778278719.311813 read 0x092d93 thru 0x096c5d (0x3ecb bytes) 000307[1] 1778278719.325055 write 0x3450b5 thru 0x34a959 (0x58a5 bytes) 000310[0] 1778278719.356001 write 0x92a4b2 thru 0x931f04 (0x7a53 bytes) 000311[0] 1778278719.382040 trunc from 0x931f05 to 0x67baba (0x2b644a bytes) 000312[1] 1778278719.435630 mapwrite 0x3d4ae9 thru 0x3e3375 (0xe88d bytes) 000313[0] 1778278719.525023 trunc from 0x67babb to 0x55d4b9 (0x11e601 bytes) 000314[1] 1778278719.550575 trunc from 0x55d4ba to 0x74c683 (0x1ef1ca bytes) 000317[1] 1778278719.623560 mapwrite 0x40a806 thru 0x413ac0 (0x92bb bytes) 000320[1] 1778278719.708163 mapwrite 0x88d546 thru 0x88dee4 (0x99f bytes) 000321[0] 1778278719.762838 read 0x12aad5 thru 0x12de48 (0x3374 bytes) 000322[0] 1778278719.774030 mapwrite 0x50520c thru 0x50d19f (0x7f94 bytes) 000323[1] 1778278719.858196 mapwrite 0x3592ac thru 0x36271d (0x9472 bytes) 000325[1] 1778278719.966043 mapread 0x0381d5 thru 0x040efc (0x8d28 bytes) 000326[1] 1778278720.023554 write 0x77855d thru 0x7798f2 (0x1396 bytes) 000327[0] 1778278720.046764 write 0x8ce715 thru 0x8de540 (0xfe2c bytes) 000328[0] 1778278720.076376 trunc from 0x8de541 to 0x705987 (0x1d8bb9 bytes) 000329[0] 1778278720.144516 mapwrite 0x71026e thru 0x71cc6d (0xca00 bytes) 000330[1] 1778278720.236592 trunc from 0x71cc6e to 0x610a73 (0x10c1fa bytes) 000332[1] 1778278720.282987 read 0x255816 thru 0x2577af (0x1f9a bytes) 000333[1] 1778278720.296336 trunc from 0x610a74 to 0x25bd50 (0x3b4d23 bytes) 000337[0] 1778278720.372408 write 0x77f57d thru 0x78cd31 (0xd7b5 bytes) 000338[0] 1778278720.407364 read 0x2e26e3 thru 0x2e7c1b (0x5539 bytes) 000340[1] 1778278720.491364 write 0x634e10 thru 0x63ae67 (0x6058 bytes) 000345[1] 1778278720.564879 mapread 0x5b7e8c thru 0x5bf3f4 (0x7569 bytes) 000349[0] 1778278720.635128 read 0x63f766 thru 0x647814 (0x80af bytes) 000350[0] 1778278720.669803 read 0x173ce0 thru 0x175a54 (0x1d75 bytes) 000352[1] 1778278720.678781 write 0x1b783e thru 0x1bd19f (0x5962 bytes) 000354[1] 1778278720.731062 mapread 0x3e18ae thru 0x3e8f0b (0x765e bytes) 000355[1] 1778278720.790163 write 0x1687c0 thru 0x16c918 (0x4159 bytes) 000356[0] 1778278720.800954 mapwrite 0x61b37f thru 0x61f7ea (0x446c bytes) 000357[0] 1778278720.883807 mapwrite 0x2473a0 thru 0x2566cb (0xf32c bytes) 000358[1] 1778278720.983884 mapread 0x34aecd thru 0x34f03e (0x4172 bytes) 000359[0] 1778278721.096196 write 0x2a13b4 thru 0x2a15cc (0x219 bytes) 000362[0] 1778278721.145390 trunc from 0x78cd32 to 0x1c508e (0x5c7ca3 bytes) 000365[0] 1778278721.165451 mapwrite 0x181b16 thru 0x190af3 (0xefde bytes) 000366[1] 1778278721.235026 write 0x45498c thru 0x4648f0 (0xff65 bytes) 000368[1] 1778278721.273469 read 0x4479f5 thru 0x44e26c (0x6878 bytes) 000369[1] 1778278721.291523 mapread 0x3458fb thru 0x34d336 (0x7a3c bytes) 000371[0] 1778278721.351540 write 0x7fb7a7 thru 0x7ffe9f (0x46f9 bytes) 000372[0] 1778278721.385246 mapread 0x2078ea thru 0x209784 (0x1e9b bytes) 000373[1] 1778278721.402408 mapread 0x3bbea1 thru 0x3c3538 (0x7698 bytes) 000375[0] 1778278721.529408 read 0x27ecfa thru 0x28e6cd (0xf9d4 bytes) 000380[1] 1778278721.604979 write 0x8c8e49 thru 0x8d1489 (0x8641 bytes) 000381[1] 1778278721.642060 mapread 0x4bbd0c thru 0x4c7b7d (0xbe72 bytes) 000384[0] 1778278721.721397 mapwrite 0x37e6a5 thru 0x388523 (0x9e7f bytes) 000386[1] 1778278721.804504 write 0x85026a thru 0x85c919 (0xc6b0 bytes) 000389[0] 1778278721.838223 mapread 0x211748 thru 0x212010 (0x8c9 bytes) 000393[0] 1778278721.863768 mapwrite 0x996877 thru 0x999d05 (0x348f bytes) 000394[0] 1778278721.938894 mapwrite 0x9e89ce thru 0x9f50bb (0xc6ee bytes) 000395[0] 1778278721.995751 mapwrite 0x4d831b thru 0x4e790a (0xf5f0 bytes) 000399[0] 1778278722.055143 write 0x75a011 thru 0x761c3d (0x7c2d bytes) 000401[0] 1778278722.087105 read 0x9c513e thru 0x9d4b06 (0xf9c9 bytes) 000402[0] 1778278722.093967 trunc from 0x9f50bc to 0x29068a (0x764a31 bytes) 000404[1] 1778278722.110927 mapread 0x24b665 thru 0x24eb9a (0x3536 bytes) 000405[0] 1778278722.151734 read 0x008902 thru 0x0142e2 (0xb9e1 bytes) 000408[1] 1778278722.236888 mapwrite 0x650cbd thru 0x65d922 (0xcc66 bytes) 000410[1] 1778278722.353381 read 0x468a2b thru 0x46fcb1 (0x7287 bytes) 000412[1] 1778278722.402225 mapread 0x3d4e2a thru 0x3db6ca (0x68a1 bytes) 000413[0] 1778278722.445831 mapwrite 0x8d871d thru 0x8e2823 (0xa107 bytes) 000414[0] 1778278722.589116 mapread 0x295f99 thru 0x29ea06 (0x8a6e bytes) 000415[1] 1778278722.603827 mapwrite 0x4f6949 thru 0x503675 (0xcd2d bytes) 000417[0] 1778278722.706865 read 0x7a8f32 thru 0x7b74c2 (0xe591 bytes) 000420[0] 1778278722.740889 read 0x70b24e thru 0x717b38 (0xc8eb bytes) 000422[1] 1778278722.766878 write 0x20df65 thru 0x2112a7 (0x3343 bytes) 000423[1] 1778278722.789120 trunc from 0x8e2824 to 0x04f26a (0x8935b9 bytes) 000425[1] 1778278722.817568 mapwrite 0x6ec6ee thru 0x6f47cd (0x80e0 bytes) 000426[0] 1778278722.896182 mapwrite 0x64a820 thru 0x64c521 (0x1d02 bytes) 000430[0] 1778278722.993988 read 0x348f54 thru 0x3506e3 (0x7790 bytes) 000431[0] 1778278723.004562 mapread 0x144c33 thru 0x14aa3f (0x5e0d bytes) 000432[1] 1778278723.023003 trunc from 0x6f47ce to 0x80d03b (0x11886e bytes) 000433[1] 1778278723.056596 trunc from 0x80d03c to 0x1d0a48 (0x63c5f3 bytes) 000434[1] 1778278723.069210 write 0x3ae29c thru 0x3b5d00 (0x7a65 bytes) 000437[1] 1778278723.100552 write 0x4d819c thru 0x4deb9c (0x6a01 bytes) 000440[1] 1778278723.136102 read 0x3f5aa6 thru 0x3ff7d6 (0x9d31 bytes) 000441[1] 1778278723.157310 mapread 0x3d3168 thru 0x3db228 (0x80c1 bytes) 000445[1] 1778278723.202460 mapwrite 0x05840f thru 0x0619dd (0x95cf bytes) 000449[1] 1778278723.268649 mapwrite 0x1130a6 thru 0x12185d (0xe7b8 bytes) 000453[1] 1778278723.393348 trunc from 0x4deb9d to 0x3bb853 (0x123349 bytes) 000455[1] 1778278723.408263 write 0x0aae2a thru 0x0b883e (0xda15 bytes) 000456[0] 1778278723.418184 trunc from 0x3bb854 to 0x0777ce (0x344085 bytes) 000457[0] 1778278723.459836 mapwrite 0x53ce43 thru 0x547ea6 (0xb064 bytes) 000459[0] 1778278723.585007 mapread 0x2c6ada thru 0x2cbbfc (0x5123 bytes) 000460[0] 1778278723.604069 write 0x4f98e4 thru 0x50770a (0xde27 bytes) 000461[0] 1778278723.611106 read 0x00dfa7 thru 0x015272 (0x72cc bytes) 000462[0] 1778278723.630639 trunc from 0x547ea7 to 0x18158a (0x3c691c bytes) 000463[1] 1778278723.646419 read 0x064fb0 thru 0x06a252 (0x52a3 bytes) 000465[0] 1778278723.680032 read 0x0d1692 thru 0x0d9beb (0x855a bytes) 000467[0] 1778278723.709037 write 0x10d695 thru 0x10db74 (0x4e0 bytes) 000469[0] 1778278723.742034 trunc from 0x18158b to 0x13e16d (0x4341d bytes) 000470[1] 1778278723.753266 mapwrite 0x92f0fb thru 0x93e746 (0xf64c bytes) 000471[0] 1778278723.846482 mapread 0x513295 thru 0x516559 (0x32c5 bytes) 000475[1] 1778278723.885514 read 0x0ea2af thru 0x0f004d (0x5d9f bytes) truncating to largest ever: 0x9dd5ed 000476[1] 1778278723.892263 trunc from 0x93e747 to 0x9dd5ec (0x9eea6 bytes) 000477[1] 1778278723.932181 mapwrite 0x6e6cdd thru 0x6f3ae1 (0xce05 bytes) 000480[0] 1778278724.038251 write 0x4f624c thru 0x4f9499 (0x324e bytes) 000481[1] 1778278724.067020 mapread 0x3e04d9 thru 0x3e7907 (0x742f bytes) 000482[0] 1778278724.093459 read 0x891c61 thru 0x8995a0 (0x7940 bytes) 000484[0] 1778278724.115949 trunc from 0x9dd5ed to 0x8ca74d (0x112e9f bytes) 000485[0] 1778278724.150494 mapread 0x460167 thru 0x46d581 (0xd41b bytes) 000490[1] 1778278724.230428 read 0x32ceb7 thru 0x337a21 (0xab6b bytes) 000491[0] 1778278724.248378 mapwrite 0x6ee773 thru 0x6f70c9 (0x8957 bytes) 000492[1] 1778278724.324806 trunc from 0x8ca74e to 0x8fa70f (0x2ffc2 bytes) 000493[0] 1778278724.361391 mapwrite 0x4cd5de thru 0x4d988d (0xc2b0 bytes) 000494[1] 1778278724.465536 mapread 0x02c726 thru 0x030e3d (0x4718 bytes) 000495[1] 1778278724.510178 trunc from 0x8fa710 to 0x0b8232 (0x8424dd bytes) 000496[1] 1778278724.531359 mapwrite 0x9689d4 thru 0x97304e (0xa67b bytes) 000497[0] 1778278724.609439 trunc from 0x97304f to 0x30a6a8 (0x6689a6 bytes) 000499[0] 1778278724.651581 mapwrite 0x8cfbdd thru 0x8d4ffc (0x5420 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8724 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x101a1 000004[0] 1778278725.181742 trunc from 00000000 to 0x0101a0 (0x101a1 bytes) 000006[0] 1778278725.224279 write 0x987000 thru 0x98b4e7 (0x44e8 bytes) 000007[1] 1778278725.237193 mapwrite 0x9ae000 thru 0x9bcc85 (0xec86 bytes) 000010[0] 1778278725.338079 mapread 0x118000 thru 0x11c26d (0x426e bytes) 000015[0] 1778278725.385461 write 0x9e6000 thru 0x9f0a35 (0xaa36 bytes) 000018[1] 1778278725.401369 mapread 0x428000 thru 0x428b98 (0xb99 bytes) 000019[0] 1778278725.429148 write 0x0b3000 thru 0x0b6032 (0x3033 bytes) 000020[0] 1778278725.449699 mapread 0x717000 thru 0x71c527 (0x5528 bytes) 000023[0] 1778278725.484452 read 0x200000 thru 0x20a66e (0xa66f bytes) 000024[0] 1778278725.504772 mapread 0x18e000 thru 0x199045 (0xb046 bytes) 000025[0] 1778278725.531755 mapwrite 0x9b9000 thru 0x9c2498 (0x9499 bytes) truncating to largest ever: 0x5b0516 000026[0] 1778278725.587105 trunc from 0x9f0a36 to 0x5b0515 (0x440520 bytes) 000027[0] 1778278725.603584 mapwrite 0x549000 thru 0x54efb9 (0x5fba bytes) 000029[0] 1778278725.680209 read 0x2d6000 thru 0x2d9899 (0x389a bytes) 000030[1] 1778278725.688014 read_OD 0x1d6000 thru 0x1defff (0x9000 bytes) 000031[0] 1778278725.711957 write 0x393000 thru 0x399b22 (0x6b23 bytes) 000033[0] 1778278725.724909 mapread 0x460000 thru 0x46d51e (0xd51f bytes) 000038[0] 1778278725.757525 mapread 0x18c000 thru 0x19975c (0xd75d bytes) 000041[1] 1778278725.777939 mapread 0x57d000 thru 0x5826df (0x56e0 bytes) 000043[0] 1778278725.817856 read 0x29b000 thru 0x2a78d2 (0xc8d3 bytes) truncating to largest ever: 0x967b33 000047[1] 1778278725.833812 trunc from 0x5b0516 to 0x967b32 (0x3b761d bytes) 000048[1] 1778278725.875970 read_OD 0x669000 thru 0x676fff (0xe000 bytes) 000049[1] 1778278725.895294 read_OD 0x366000 thru 0x374fff (0xf000 bytes) 000051[0] 1778278725.911393 trunc from 0x967b33 to 0x60221b (0x365917 bytes) 000052[0] 1778278725.947124 read 0x0bc000 thru 0x0bf2b9 (0x32ba bytes) 000053[0] 1778278725.952919 mapwrite 0x7b3000 thru 0x7c2282 (0xf283 bytes) 000054[0] 1778278726.036510 trunc from 0x7c2283 to 0x8d225f (0x10ffdd bytes) 000055[1] 1778278726.052996 mapwrite 0x0e3000 thru 0x0e9ec4 (0x6ec5 bytes) 000057[1] 1778278726.111802 trunc from 0x8d2260 to 0x73ef9a (0x1932c5 bytes) 000060[1] 1778278726.126791 mapread 0x6ea000 thru 0x6f6071 (0xc072 bytes) 000061[1] 1778278726.154077 mapread 0x46b000 thru 0x4735dc (0x85dd bytes) 000062[0] 1778278726.181893 trunc from 0x73ef9b to 0x10acb6 (0x6342e4 bytes) 000064[0] 1778278726.205894 read 0x108000 thru 0x108d22 (0xd23 bytes) 000065[0] 1778278726.220483 write 0x695000 thru 0x69b0d6 (0x60d7 bytes) 000066[0] 1778278726.222866 write 0x850000 thru 0x85e854 (0xe855 bytes) 000068[0] 1778278726.238343 mapwrite 0x6ed000 thru 0x6f8f6f (0xbf70 bytes) 000070[0] 1778278726.315382 write 0x36f000 thru 0x37235e (0x335f bytes) 000071[0] 1778278726.338346 trunc from 0x85e855 to 0x36b4ba (0x4f339a bytes) 000072[0] 1778278726.387269 mapread 0x0b5000 thru 0x0c452e (0xf52f bytes) 000074[0] 1778278726.407724 mapread 0x2f5000 thru 0x303be3 (0xebe4 bytes) 000075[0] 1778278726.408705 trunc from 0x36b4bb to 0x2e5f8f (0x8552b bytes) 000076[1] 1778278726.436845 read_OD 0x25b000 thru 0x264fff (0xa000 bytes) 000077[0] 1778278726.469296 trunc from 0x2e5f90 to 0x27075e (0x75831 bytes) 000078[0] 1778278726.487583 mapread 0x1ca000 thru 0x1d2e13 (0x8e14 bytes) 000081[1] 1778278726.529489 mapread 0x0a5000 thru 0x0adc44 (0x8c45 bytes) 000082[0] 1778278726.575657 read 0x1a8000 thru 0x1af407 (0x7408 bytes) 000084[0] 1778278726.601989 mapwrite 0x37e000 thru 0x384883 (0x6884 bytes) 000085[1] 1778278726.706546 write_OD 0x253000 thru 0x256fff (0x4000 bytes) 000086[0] 1778278726.765518 trunc from 0x384884 to 0x3f5f2b (0x716a8 bytes) 000087[1] 1778278726.789848 mapwrite 0x990000 thru 0x99c4fc (0xc4fd bytes) 000089[0] 1778278726.941711 read 0x436000 thru 0x445092 (0xf093 bytes) 000090[1] 1778278726.973287 trunc from 0x99c4fd to 0x88bf64 (0x110598 bytes) 000091[0] 1778278726.995482 mapread 0x228000 thru 0x2340b5 (0xc0b6 bytes) 000094[1] 1778278727.037951 mapread 0x22c000 thru 0x23173e (0x573f bytes) 000096[1] 1778278727.052145 mapwrite 0x097000 thru 0x0a4748 (0xd749 bytes) 000098[0] 1778278727.150588 trunc from 0x88bf65 to 0x7bced7 (0xcf08d bytes) 000099[1] 1778278727.180476 trunc from 0x7bced8 to 0x1e8eb5 (0x5d4022 bytes) 000101[0] 1778278727.218230 trunc from 0x1e8eb6 to 0x7c496b (0x5dbab6 bytes) 000104[1] 1778278727.288869 mapwrite 0x609000 thru 0x611005 (0x8006 bytes) 000105[0] 1778278727.366506 mapread 0x694000 thru 0x6a3b35 (0xfb36 bytes) 000107[1] 1778278727.400079 write_OD 0x700000 thru 0x706fff (0x7000 bytes) 000109[0] 1778278727.436463 read 0x4df000 thru 0x4e89f9 (0x99fa bytes) 000110[1] 1778278727.441737 write_OD 0x420000 thru 0x42efff (0xf000 bytes) 000118[0] 1778278727.481318 write 0x61a000 thru 0x626b38 (0xcb39 bytes) 000121[0] 1778278727.511206 mapwrite 0x0d2000 thru 0x0e102b (0xf02c bytes) 000122[0] 1778278727.560100 mapread 0x455000 thru 0x45dafa (0x8afb bytes) 000123[1] 1778278727.580065 mapwrite 0x43b000 thru 0x447640 (0xc641 bytes) 000126[1] 1778278727.708540 trunc from 0x7c496c to 0x416cef (0x3adc7c bytes) 000127[0] 1778278727.737300 read 0x144000 thru 0x1498ce (0x58cf bytes) 000128[0] 1778278727.756939 mapwrite 0x68e000 thru 0x697c82 (0x9c83 bytes) 000129[1] 1778278727.834375 trunc from 0x697c83 to 0x4eb354 (0x1ac92e bytes) 000131[1] 1778278727.870203 read_OD 0x4a0000 thru 0x4a4fff (0x5000 bytes) 000134[0] 1778278727.890296 trunc from 0x4eb355 to 0x81aae2 (0x32f78e bytes) 000135[1] 1778278727.922176 mapread 0x4a0000 thru 0x4a21ba (0x21bb bytes) 000136[0] 1778278727.948916 write 0x84c000 thru 0x84f565 (0x3566 bytes) 000137[1] 1778278727.958214 mapread 0x533000 thru 0x5352c1 (0x22c2 bytes) 000138[1] 1778278727.992968 mapread 0x35d000 thru 0x3663be (0x93bf bytes) 000139[0] 1778278728.024300 mapwrite 0x5a4000 thru 0x5b15f5 (0xd5f6 bytes) 000140[0] 1778278728.106442 mapwrite 0x5b0000 thru 0x5bb24b (0xb24c bytes) 000141[0] 1778278728.186581 mapread 0x2f9000 thru 0x303686 (0xa687 bytes) 000145[0] 1778278728.225265 mapwrite 0x29b000 thru 0x2a8b8d (0xdb8e bytes) 000147[1] 1778278728.276494 write_OD 0x470000 thru 0x470fff (0x1000 bytes) 000149[1] 1778278728.310378 read_OD 0x83c000 thru 0x848fff (0xd000 bytes) 000150[0] 1778278728.314317 write 0x179000 thru 0x17cb92 (0x3b93 bytes) 000151[1] 1778278728.332791 read_OD 0x7c2000 thru 0x7cffff (0xe000 bytes) 000154[0] 1778278728.365025 mapread 0x122000 thru 0x12d0e0 (0xb0e1 bytes) 000157[0] 1778278728.381776 mapwrite 0x9af000 thru 0x9be778 (0xf779 bytes) 000158[0] 1778278728.457761 read 0x4f7000 thru 0x4f800f (0x1010 bytes) 000160[0] 1778278728.477562 mapwrite 0x843000 thru 0x84c857 (0x9858 bytes) 000162[1] 1778278728.554872 mapread 0x8c1000 thru 0x8c75cf (0x65d0 bytes) 000163[1] 1778278728.588625 write_OD 0x25d000 thru 0x26afff (0xe000 bytes) 000164[0] 1778278728.616498 mapread 0x08f000 thru 0x09c62a (0xd62b bytes) 000165[1] 1778278728.639940 mapread 0x924000 thru 0x9304a1 (0xc4a2 bytes) 000167[0] 1778278728.652674 write 0x5e8000 thru 0x5f7e42 (0xfe43 bytes) 000169[0] 1778278728.678926 mapread 0x7ff000 thru 0x804871 (0x5872 bytes) 000170[1] 1778278728.700065 read_OD 0x99a000 thru 0x9a6fff (0xd000 bytes) 000171[1] 1778278728.722181 write_OD 0x5a4000 thru 0x5acfff (0x9000 bytes) 000176[1] 1778278728.769505 write_OD 0x8dc000 thru 0x8e3fff (0x8000 bytes) 000178[0] 1778278728.818073 write 0x2dc000 thru 0x2e7741 (0xb742 bytes) 000180[1] 1778278728.852543 mapread 0x0e1000 thru 0x0e2608 (0x1609 bytes) 000181[1] 1778278728.882285 mapwrite 0x1c6000 thru 0x1c8d7e (0x2d7f bytes) 000182[1] 1778278728.940192 write_OD 0x4d2000 thru 0x4dafff (0x9000 bytes) 000184[1] 1778278728.983971 mapwrite 0x9f2000 thru 0x9ffe91 (0xde92 bytes) 000186[1] 1778278729.113784 mapwrite 0x56e000 thru 0x574aad (0x6aae bytes) 000188[1] 1778278729.163706 write_OD 0x3ac000 thru 0x3b1fff (0x6000 bytes) 000194[1] 1778278729.192545 trunc from 0x9ffe92 to 0x189fdf (0x875eb2 bytes) 000195[1] 1778278729.205549 mapread 0x0dd000 thru 0x0e18ce (0x48cf bytes) 000196[0] 1778278729.216850 trunc from 0x189fe0 to 0x51b92d (0x39194e bytes) 000197[0] 1778278729.239185 read 0x09f000 thru 0x0a6879 (0x787a bytes) 000198[1] 1778278729.248829 trunc from 0x51b92e to 0x109720 (0x41220d bytes) 000199[0] 1778278729.280229 read 0x036000 thru 0x043c0f (0xdc10 bytes) 000200[1] 1778278729.303974 mapwrite 0x5ba000 thru 0x5bebec (0x4bed bytes) 000202[1] 1778278729.384505 mapwrite 0x426000 thru 0x430f50 (0xaf51 bytes) 000205[0] 1778278729.463378 read 0x5b3000 thru 0x5b440d (0x140e bytes) 000206[1] 1778278729.469224 write_OD 0x524000 thru 0x531fff (0xe000 bytes) 000207[1] 1778278729.497897 trunc from 0x5bebed to 0x200270 (0x3be97c bytes) 000208[0] 1778278729.525417 read 0x06f000 thru 0x072099 (0x309a bytes) 000211[1] 1778278729.553273 write_OD 0x4a6000 thru 0x4a6fff (0x1000 bytes) 000213[1] 1778278729.578386 mapread 0x197000 thru 0x1a640c (0xf40d bytes) 000214[1] 1778278729.594678 write_OD 0x0cd000 thru 0x0cffff (0x3000 bytes) 000215[0] 1778278729.616684 write 0x654000 thru 0x65574b (0x174c bytes) 000217[0] 1778278729.641924 mapread 0x537000 thru 0x53edac (0x7dad bytes) 000219[0] 1778278729.703808 mapwrite 0x677000 thru 0x685ae4 (0xeae5 bytes) 000221[0] 1778278729.762928 write 0x7de000 thru 0x7e3fb8 (0x5fb9 bytes) 000223[0] 1778278729.771111 mapwrite 0x54a000 thru 0x55210f (0x8110 bytes) 000224[0] 1778278729.822767 mapwrite 0x0ab000 thru 0x0badb5 (0xfdb6 bytes) 000225[0] 1778278729.914727 mapread 0x155000 thru 0x15696b (0x196c bytes) 000226[1] 1778278729.926703 trunc from 0x7e3fb9 to 0x58b780 (0x258838 bytes) 000227[1] 1778278729.957766 write_OD 0x75f000 thru 0x764fff (0x6000 bytes) 000228[0] 1778278729.990388 mapread 0x5c1000 thru 0x5c78ee (0x68ef bytes) 000232[0] 1778278730.015498 mapwrite 0x402000 thru 0x402105 (0x106 bytes) 000234[0] 1778278730.051234 write 0x63f000 thru 0x64b1d8 (0xc1d9 bytes) 000236[0] 1778278730.064915 read 0x520000 thru 0x521a6c (0x1a6d bytes) 000237[0] 1778278730.069660 mapwrite 0x67b000 thru 0x687ccd (0xccce bytes) 000239[0] 1778278730.117001 mapread 0x0b6000 thru 0x0bbae4 (0x5ae5 bytes) 000244[0] 1778278730.127578 trunc from 0x765000 to 0x03dc90 (0x72736f bytes) 000245[1] 1778278730.144862 mapwrite 0x95f000 thru 0x96d21c (0xe21d bytes) 000248[1] 1778278730.240001 read_OD 0x436000 thru 0x43cfff (0x7000 bytes) 000252[1] 1778278730.258829 mapwrite 0x4d3000 thru 0x4d358e (0x58f bytes) 000253[0] 1778278730.308382 write 0x53d000 thru 0x5454f3 (0x84f4 bytes) 000254[1] 1778278730.330364 mapwrite 0x85f000 thru 0x86547a (0x647b bytes) 000257[0] 1778278730.430431 read 0x6d1000 thru 0x6d4c46 (0x3c47 bytes) 000261[1] 1778278730.440966 read_OD 0x339000 thru 0x339fff (0x1000 bytes) 000264[1] 1778278730.443944 read_OD 0x0a9000 thru 0x0b3fff (0xb000 bytes) 000267[1] 1778278730.448556 read_OD 0x6c8000 thru 0x6cdfff (0x6000 bytes) 000268[0] 1778278730.452941 trunc from 0x96d21d to 0x944ca3 (0x28579 bytes) 000269[1] 1778278730.480428 mapwrite 0x4cb000 thru 0x4cd0f9 (0x20fa bytes) 000270[1] 1778278730.533208 mapread 0x419000 thru 0x427675 (0xe676 bytes) 000273[0] 1778278730.550863 mapwrite 0x30d000 thru 0x31c001 (0xf002 bytes) 000274[1] 1778278730.618859 write_OD 0x727000 thru 0x730fff (0xa000 bytes) 000276[0] 1778278730.648716 mapread 0x534000 thru 0x538f72 (0x4f73 bytes) 000278[1] 1778278730.658847 read_OD 0x099000 thru 0x0a7fff (0xf000 bytes) 000279[1] 1778278730.665650 write_OD 0x01c000 thru 0x022fff (0x7000 bytes) 000282[1] 1778278730.694143 mapwrite 0x927000 thru 0x92afb4 (0x3fb5 bytes) 000283[1] 1778278730.749394 mapread 0x77d000 thru 0x787778 (0xa779 bytes) 000284[1] 1778278730.764353 trunc from 0x944ca4 to 0x759868 (0x1eb43b bytes) 000286[1] 1778278730.774420 mapwrite 0x1f7000 thru 0x2031fe (0xc1ff bytes) 000289[1] 1778278730.820745 trunc from 0x759869 to 0x0ae6ef (0x6ab179 bytes) 000291[1] 1778278730.836947 mapwrite 0x11d000 thru 0x11eaec (0x1aed bytes) 000295[1] 1778278730.893422 write_OD 0x95c000 thru 0x95dfff (0x2000 bytes) 000302[0] 1778278730.939963 read 0x4c3000 thru 0x4ca282 (0x7283 bytes) 000305[0] 1778278730.946692 mapread 0x2e0000 thru 0x2e7c66 (0x7c67 bytes) 000306[0] 1778278730.960241 read 0x463000 thru 0x46dea7 (0xaea8 bytes) 000307[0] 1778278730.965306 trunc from 0x95e000 to 0x4e4932 (0x4796cd bytes) 000312[1] 1778278731.019591 read_OD 0x31d000 thru 0x31dfff (0x1000 bytes) 000315[1] 1778278731.053214 trunc from 0x4e4933 to 0x0a2aea (0x441e48 bytes) 000320[1] 1778278731.092400 mapread 0x064000 thru 0x072097 (0xe098 bytes) 000322[1] 1778278731.127596 write_OD 0x10d000 thru 0x111fff (0x5000 bytes) 000325[1] 1778278731.160196 mapread 0x010000 thru 0x01056c (0x56d bytes) 000327[0] 1778278731.164363 trunc from 0x112000 to 0x0ed778 (0x24887 bytes) 000331[1] 1778278731.208568 read 0x0c9000 thru 0x0d5098 (0xc099 bytes) 000332[1] 1778278731.219054 mapwrite 0x129000 thru 0x137057 (0xe058 bytes) 000333[1] 1778278731.277336 read 0x05d000 thru 0x06bbf6 (0xebf7 bytes) 000336[0] 1778278731.292076 mapread 0x0dc000 thru 0x0e80eb (0xc0ec bytes) 000340[0] 1778278731.335373 mapread 0x081000 thru 0x084bac (0x3bad bytes) 000343[0] 1778278731.341136 mapread 0x0e1000 thru 0x0ed82b (0xc82c bytes) 000350[0] 1778278731.343892 mapwrite 0x117000 thru 0x118b55 (0x1b56 bytes) 000354[1] 1778278731.409757 mapread 0x0e7000 thru 0x0e9138 (0x2139 bytes) 000359[0] 1778278731.427114 read 0x11b000 thru 0x11ce55 (0x1e56 bytes) 000360[1] 1778278731.437872 mapread 0x06f000 thru 0x0727e0 (0x37e1 bytes) 000361[0] 1778278731.455259 mapread 0x128000 thru 0x136237 (0xe238 bytes) 000364[0] 1778278731.459230 mapwrite 0x43c000 thru 0x448773 (0xc774 bytes) 000366[0] 1778278731.567944 mapwrite 0x1b9000 thru 0x1b9530 (0x531 bytes) 000367[0] 1778278731.591936 read 0x234000 thru 0x2364ff (0x2500 bytes) 000368[1] 1778278731.598054 read 0x2cc000 thru 0x2cdbb4 (0x1bb5 bytes) 000369[1] 1778278731.609859 mapwrite 0x1fe000 thru 0x20589e (0x789f bytes) 000370[1] 1778278731.656148 write 0x160000 thru 0x16c865 (0xc866 bytes) 000371[0] 1778278731.659502 write 0x65c000 thru 0x6630bf (0x70c0 bytes) 000373[0] 1778278731.681771 trunc from 0x6630c0 to 0x01fbae (0x643511 bytes) 000375[1] 1778278731.717955 trunc from 0x01fbaf to 0x1b9407 (0x199859 bytes) 000377[1] 1778278731.742848 mapwrite 0x463000 thru 0x4675dc (0x45dd bytes) 000379[0] 1778278731.811793 trunc from 0x4675dd to 0x6e1b85 (0x27a5a9 bytes) 000382[1] 1778278731.848906 write 0x23d000 thru 0x240a8c (0x3a8d bytes) 000386[1] 1778278731.877172 mapread 0x2ee000 thru 0x2f0175 (0x2176 bytes) 000387[1] 1778278731.905325 mapwrite 0x430000 thru 0x431bd7 (0x1bd8 bytes) 000390[0] 1778278731.959236 write 0x86e000 thru 0x87b378 (0xd379 bytes) 000391[0] 1778278731.968898 write 0x11f000 thru 0x120596 (0x1597 bytes) 000392[1] 1778278731.987148 mapwrite 0x23f000 thru 0x24132f (0x2330 bytes) 000393[1] 1778278732.031788 write 0x61d000 thru 0x62b2e1 (0xe2e2 bytes) 000394[1] 1778278732.036717 trunc from 0x87b379 to 0x541785 (0x339bf3 bytes) 000400[1] 1778278732.063870 mapwrite 0x94e000 thru 0x95a085 (0xc086 bytes) 000401[1] 1778278732.156737 write 0x1e9000 thru 0x1f5171 (0xc172 bytes) 000402[0] 1778278732.182218 mapread 0x5ba000 thru 0x5baafc (0xafd bytes) 000404[0] 1778278732.207757 mapwrite 0x9e1000 thru 0x9e3a69 (0x2a6a bytes) 000407[1] 1778278732.319535 mapread 0x1b7000 thru 0x1bd4bc (0x64bd bytes) 000408[1] 1778278732.350192 write 0x50d000 thru 0x51bcc9 (0xecca bytes) 000409[1] 1778278732.368162 trunc from 0x9e3a6a to 0x6588ed (0x38b17c bytes) 000411[0] 1778278732.389825 mapread 0x598000 thru 0x59ab65 (0x2b66 bytes) 000412[0] 1778278732.413579 trunc from 0x6588ee to 0x912f82 (0x2ba695 bytes) 000415[0] 1778278732.452219 mapwrite 0x969000 thru 0x976a1b (0xda1c bytes) 000416[0] 1778278732.520569 read 0x0ec000 thru 0x0f3418 (0x7419 bytes) 000419[0] 1778278732.558760 write 0x976000 thru 0x977a33 (0x1a34 bytes) 000421[1] 1778278732.559918 write 0x8ad000 thru 0x8af711 (0x2712 bytes) 000423[1] 1778278732.588447 mapread 0x373000 thru 0x3762d1 (0x32d2 bytes) 000425[1] 1778278732.614129 write 0x995000 thru 0x995b65 (0xb66 bytes) 000426[0] 1778278732.625730 write 0x43b000 thru 0x4428e7 (0x78e8 bytes) 000432[0] 1778278732.656388 mapwrite 0x738000 thru 0x738e31 (0xe32 bytes) 000434[1] 1778278732.718781 mapwrite 0x35c000 thru 0x369cb3 (0xdcb4 bytes) 000435[1] 1778278732.788854 write 0x7f8000 thru 0x807962 (0xf963 bytes) 000436[0] 1778278732.795996 read 0x794000 thru 0x79b273 (0x7274 bytes) 000437[1] 1778278732.823804 write 0x1a1000 thru 0x1a379f (0x27a0 bytes) 000444[1] 1778278732.900205 read 0x4d2000 thru 0x4d5028 (0x3029 bytes) 000446[1] 1778278732.942231 write 0x23d000 thru 0x242138 (0x5139 bytes) 000447[1] 1778278732.947809 mapwrite 0x732000 thru 0x7352a6 (0x32a7 bytes) 000455[0] 1778278733.053167 trunc from 0x995b66 to 0x5724e6 (0x42367f bytes) 000456[0] 1778278733.085824 trunc from 0x5724e7 to 0x1b9588 (0x3b8f5e bytes) 000457[1] 1778278733.103762 write 0x3d4000 thru 0x3e2775 (0xe776 bytes) 000463[0] 1778278733.137326 read 0x259000 thru 0x25d22a (0x422b bytes) 000464[0] 1778278733.187272 mapread 0x11c000 thru 0x11ca24 (0xa25 bytes) 000465[0] 1778278733.190193 trunc from 0x3e2776 to 0x48246f (0x9fcfa bytes) 000467[0] 1778278733.243921 mapwrite 0x067000 thru 0x06e249 (0x724a bytes) 000469[0] 1778278733.325035 read 0x29c000 thru 0x2a843c (0xc43d bytes) 000470[0] 1778278733.329329 mapread 0x0cf000 thru 0x0d3c67 (0x4c68 bytes) 000471[0] 1778278733.353342 mapwrite 0x965000 thru 0x96c4d4 (0x74d5 bytes) 000472[0] 1778278733.419950 trunc from 0x96c4d5 to 0x7e5276 (0x18725e bytes) 000475[1] 1778278733.447847 mapwrite 0x5a9000 thru 0x5b54d7 (0xc4d8 bytes) 000478[1] 1778278733.531087 mapread 0x1d3000 thru 0x1de1fa (0xb1fb bytes) 000479[0] 1778278733.570397 read 0x151000 thru 0x156a7b (0x5a7c bytes) 000480[1] 1778278733.582325 mapread 0x4a0000 thru 0x4a2f48 (0x2f49 bytes) 000481[0] 1778278733.608731 read 0x1c8000 thru 0x1ccb1c (0x4b1d bytes) 000483[1] 1778278733.617481 mapread 0x5d7000 thru 0x5dbd08 (0x4d09 bytes) 000484[0] 1778278733.645070 read 0x001000 thru 0x009b2c (0x8b2d bytes) 000485[0] 1778278733.655824 mapread 0x0c9000 thru 0x0c9678 (0x679 bytes) 000488[0] 1778278733.660956 mapread 0x331000 thru 0x33b0fd (0xa0fe bytes) 000489[0] 1778278733.680386 trunc from 0x7e5277 to 0x3378b0 (0x4ad9c6 bytes) 000490[1] 1778278733.707328 read 0x0c6000 thru 0x0d4c39 (0xec3a bytes) 000491[1] 1778278733.721666 read 0x082000 thru 0x08d01a (0xb01b bytes) 000494[1] 1778278733.748206 read 0x0c9000 thru 0x0d3e9a (0xae9b bytes) 000495[1] 1778278733.748629 write 0x3e9000 thru 0x3f48b5 (0xb8b6 bytes) 000496[0] 1778278733.760214 write 0x7a3000 thru 0x7a88a2 (0x58a3 bytes) 000497[1] 1778278733.771398 write 0x9fc000 thru 0x9ffb70 (0x3b71 bytes) 000498[0] 1778278733.784781 write 0x09b000 thru 0x0a0e5e (0x5e5f bytes) 000500[1] 1778278733.806086 read 0x84e000 thru 0x84e70c (0x70d bytes) All operations completed A-OK! PASS 16a (29s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 18:18:56 (1778278736) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.1159 s, 72.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.136309 s, 61.5 MB/s Chance of close/open is 1 in 50 Seed set to 8737 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000006[0] 1778278737.861974 write 0x414bb3 thru 0x416441 (0x188f bytes) 000011[0] 1778278737.898420 mapwrite 0x9553da thru 0x9561ad (0xdd4 bytes) 000012[1] 1778278737.961586 read 0x1499db thru 0x14b3fb (0x1a21 bytes) truncating to largest ever: 0x90a2fd 000018[0] 1778278737.986008 trunc from 0x9561ae to 0x90a2fc (0x4beb1 bytes) 000019[1] 1778278738.009919 mapread 0x01e4d5 thru 0x01f1d6 (0xd02 bytes) 000022[1] 1778278738.051753 trunc from 0x90a2fd to 0x7cef4f (0x13b3ad bytes) 000023[1] 1778278738.095499 write 0x332f8a thru 0x334973 (0x19ea bytes) 000028[0] 1778278738.134676 read 0x4c3791 thru 0x4c51f2 (0x1a62 bytes) 000031[1] 1778278738.150290 mapread 0x0d6ac8 thru 0x0d6fd2 (0x50b bytes) 000032[0] 1778278738.152768 trunc from 0x7cef50 to 0x24f2de (0x57fc71 bytes) 000033[1] 1778278738.178846 read 0x034ada thru 0x036063 (0x158a bytes) 000034[0] 1778278738.196943 mapwrite 0x0228ab thru 0x023ae4 (0x123a bytes) 000039[0] 1778278738.287296 write 0x582472 thru 0x5830d4 (0xc63 bytes) 000041[1] 1778278738.302222 mapwrite 0x79fb47 thru 0x79fceb (0x1a5 bytes) 000043[0] 1778278738.399812 mapread 0x0c24fd thru 0x0c41df (0x1ce3 bytes) 000044[0] 1778278738.431726 read 0x4d3763 thru 0x4d4899 (0x1137 bytes) 000045[1] 1778278738.474354 write 0x037b58 thru 0x038dbe (0x1267 bytes) 000047[1] 1778278738.494453 mapwrite 0x011096 thru 0x012c97 (0x1c02 bytes) 000049[1] 1778278738.554127 mapread 0x3be9c7 thru 0x3bfe8c (0x14c6 bytes) 000050[0] 1778278738.581105 write 0x051cf9 thru 0x052df1 (0x10f9 bytes) 000051[0] 1778278738.601794 mapread 0x1399b8 thru 0x13b798 (0x1de1 bytes) 000052[1] 1778278738.611722 write 0x38acdb thru 0x38b0d8 (0x3fe bytes) 000053[0] 1778278738.630798 write 0x9958b4 thru 0x9978aa (0x1ff7 bytes) 000055[1] 1778278738.650210 mapread 0x6609a8 thru 0x661869 (0xec2 bytes) 000056[1] 1778278738.670037 read 0x4d56cd thru 0x4d6beb (0x151f bytes) 000057[0] 1778278738.681019 read 0x5e400a thru 0x5e4866 (0x85d bytes) 000059[1] 1778278738.696675 write 0x1722c0 thru 0x173ae3 (0x1824 bytes) 000060[0] 1778278738.709887 trunc from 0x9978ab to 0x2839ce (0x713edc bytes) 000063[1] 1778278738.756837 trunc from 0x2839cf to 0x3fbd45 (0x178377 bytes) 000068[0] 1778278738.818587 read 0x08ec3f thru 0x08ee2c (0x1ee bytes) 000070[1] 1778278738.872194 write 0x7d803e thru 0x7d837d (0x340 bytes) 000073[0] 1778278738.874818 trunc from 0x7d837e to 0x2eb458 (0x4ecf25 bytes) 000074[1] 1778278738.917188 trunc from 0x2eb459 to 0x6bd705 (0x3d22ad bytes) 000075[0] 1778278738.933878 mapwrite 0x1761be thru 0x177aca (0x190d bytes) 000076[1] 1778278738.983846 read 0x075e93 thru 0x076915 (0xa83 bytes) 000079[0] 1778278739.017492 mapwrite 0x91bb4d thru 0x91d5a3 (0x1a57 bytes) 000081[1] 1778278739.126331 read 0x46f91b thru 0x470a5f (0x1145 bytes) 000082[1] 1778278739.169682 mapwrite 0x31ee66 thru 0x31f542 (0x6dd bytes) 000085[1] 1778278739.238615 write 0x9b3825 thru 0x9b40b2 (0x88e bytes) 000089[1] 1778278739.251473 read 0x254128 thru 0x255659 (0x1532 bytes) 000090[1] 1778278739.254550 trunc from 0x9b40b3 to 0x029ec4 (0x98a1ee bytes) 000094[1] 1778278739.309071 write 0x0da801 thru 0x0dbf08 (0x1708 bytes) 000096[1] 1778278739.346585 write 0x6f7ff6 thru 0x6f8048 (0x53 bytes) 000099[1] 1778278739.396480 mapwrite 0x68bcfc thru 0x68ca11 (0xd16 bytes) 000101[0] 1778278739.443493 read 0x116f84 thru 0x116fcb (0x48 bytes) 000102[0] 1778278739.484927 mapread 0x51645b thru 0x516647 (0x1ed bytes) 000104[0] 1778278739.509176 mapwrite 0x20ff8a thru 0x210b4a (0xbc1 bytes) 000106[1] 1778278739.590315 read 0x6ba2dd thru 0x6ba4b1 (0x1d5 bytes) 000107[0] 1778278739.594138 read 0x5c71a7 thru 0x5c7fdd (0xe37 bytes) 000108[0] 1778278739.605260 mapwrite 0x1fdda7 thru 0x1ff98a (0x1be4 bytes) 000109[1] 1778278739.650728 mapread 0x6971ca thru 0x69735e (0x195 bytes) 000115[1] 1778278739.706854 read 0x0b97e0 thru 0x0b9fb4 (0x7d5 bytes) 000117[1] 1778278739.717172 mapread 0x1294f0 thru 0x129bfa (0x70b bytes) 000121[1] 1778278739.723793 mapwrite 0x27a623 thru 0x27c171 (0x1b4f bytes) 000124[0] 1778278739.766093 trunc from 0x6f8049 to 0x8bddc8 (0x1c5d80 bytes) 000125[1] 1778278739.786500 mapread 0x0d9739 thru 0x0db2fd (0x1bc5 bytes) 000128[1] 1778278739.809811 write 0x552b0b thru 0x553958 (0xe4e bytes) 000132[1] 1778278739.834117 mapread 0x774703 thru 0x775d40 (0x163e bytes) 000133[1] 1778278739.853450 trunc from 0x8bddc9 to 0x7c60f6 (0xf7cd2 bytes) 000135[1] 1778278739.898050 write 0x094529 thru 0x094eea (0x9c2 bytes) 000136[0] 1778278739.920817 mapwrite 0x7ef73b thru 0x7f0802 (0x10c8 bytes) 000138[1] 1778278740.028466 trunc from 0x7f0803 to 0x27073a (0x5800c8 bytes) 000139[0] 1778278740.054372 mapread 0x0426da thru 0x042c83 (0x5aa bytes) 000141[0] 1778278740.112650 mapwrite 0x5d5cb1 thru 0x5d79ab (0x1cfb bytes) 000143[0] 1778278740.187492 write 0x8e6d08 thru 0x8e7b27 (0xe20 bytes) 000145[0] 1778278740.192444 write 0x4302a9 thru 0x430ff6 (0xd4e bytes) 000146[1] 1778278740.195567 mapread 0x44fa00 thru 0x45090f (0xf10 bytes) 000150[1] 1778278740.240099 trunc from 0x8e7b28 to 0x627800 (0x2c0327 bytes) 000151[1] 1778278740.275873 mapwrite 0x84c2f8 thru 0x84ca89 (0x792 bytes) 000153[1] 1778278740.323347 read 0x0a0590 thru 0x0a10df (0xb50 bytes) 000154[1] 1778278740.328811 read 0x652a07 thru 0x65470d (0x1d07 bytes) 000156[1] 1778278740.366907 write 0x96d6ad thru 0x96e98e (0x12e2 bytes) 000160[1] 1778278740.381958 mapwrite 0x3fbc7f thru 0x3fd102 (0x1484 bytes) 000162[0] 1778278740.436865 read 0x523f41 thru 0x5250af (0x116f bytes) 000166[1] 1778278740.478294 write 0x6806b7 thru 0x6813c9 (0xd13 bytes) 000169[1] 1778278740.536287 mapread 0x4bab44 thru 0x4bc9f2 (0x1eaf bytes) 000172[0] 1778278740.573077 write 0x13f4f2 thru 0x1403dc (0xeeb bytes) 000175[0] 1778278740.657543 read 0x4674a0 thru 0x4684a3 (0x1004 bytes) 000178[0] 1778278740.689202 read 0x27fee2 thru 0x2800c1 (0x1e0 bytes) 000179[1] 1778278740.694293 read 0x8d58f5 thru 0x8d77b7 (0x1ec3 bytes) 000182[0] 1778278740.714301 write 0x4759ff thru 0x475a61 (0x63 bytes) 000185[1] 1778278740.760094 write 0x1dfd2c thru 0x1e12d3 (0x15a8 bytes) 000186[1] 1778278740.791542 mapwrite 0x388443 thru 0x389e16 (0x19d4 bytes) 000188[1] 1778278740.884688 trunc from 0x96e98f to 0x2db131 (0x69385d bytes) 000191[1] 1778278740.922125 trunc from 0x2db132 to 0x08c78a (0x24e9a7 bytes) 000193[0] 1778278740.954332 write 0x778fa4 thru 0x77aa27 (0x1a84 bytes) 000194[1] 1778278740.975454 mapwrite 0x814dd5 thru 0x816076 (0x12a2 bytes) 000197[0] 1778278741.038303 trunc from 0x816077 to 0x10701b (0x70f05b bytes) 000198[1] 1778278741.053955 trunc from 0x10701c to 0x6ec300 (0x5e52e5 bytes) 000200[0] 1778278741.082144 mapread 0x3057d7 thru 0x306f6a (0x1794 bytes) 000203[0] 1778278741.119584 read 0x1c464a thru 0x1c4dfb (0x7b2 bytes) 000204[0] 1778278741.127540 write 0x0d93a7 thru 0x0db18c (0x1de6 bytes) 000205[1] 1778278741.150300 mapwrite 0x926691 thru 0x9274da (0xe4a bytes) 000208[0] 1778278741.207981 mapread 0x1fd042 thru 0x1fd402 (0x3c1 bytes) 000209[0] 1778278741.227508 mapwrite 0x1b704d thru 0x1b7413 (0x3c7 bytes) 000210[1] 1778278741.266371 mapread 0x779f83 thru 0x77ae04 (0xe82 bytes) 000215[0] 1778278741.326042 trunc from 0x9274db to 0x4d1319 (0x4561c1 bytes) 000218[0] 1778278741.360518 write 0x778114 thru 0x779745 (0x1632 bytes) 000220[1] 1778278741.379592 mapread 0x1113c0 thru 0x11333a (0x1f7b bytes) 000224[0] 1778278741.439143 mapwrite 0x04b02d thru 0x04c1f2 (0x11c6 bytes) 000226[1] 1778278741.494153 read 0x2c8083 thru 0x2c89a9 (0x927 bytes) 000227[0] 1778278741.519216 mapread 0x2d396d thru 0x2d3c1d (0x2b1 bytes) 000229[0] 1778278741.525061 write 0x3cdc68 thru 0x3ced2a (0x10c3 bytes) 000230[0] 1778278741.565302 mapwrite 0x476f70 thru 0x477f74 (0x1005 bytes) 000231[1] 1778278741.630253 mapread 0x3d317b thru 0x3d510b (0x1f91 bytes) 000234[0] 1778278741.704194 read 0x4f854c thru 0x4f886e (0x323 bytes) 000235[1] 1778278741.727164 trunc from 0x779746 to 0x4e357e (0x2961c7 bytes) 000237[0] 1778278741.800680 mapread 0x10607d thru 0x10760a (0x158e bytes) 000239[1] 1778278741.851926 trunc from 0x4e357f to 0x8b2bc2 (0x3cf644 bytes) 000241[1] 1778278741.904868 read 0x3064de thru 0x3082a8 (0x1dcb bytes) 000243[0] 1778278741.923878 mapread 0x30f33b thru 0x30fdd0 (0xa96 bytes) 000244[0] 1778278741.942339 write 0x78865c thru 0x7895eb (0xf90 bytes) 000245[0] 1778278741.969135 mapwrite 0x9e098d thru 0x9e1902 (0xf76 bytes) 000248[1] 1778278742.044956 read 0x4665d1 thru 0x466a6c (0x49c bytes) 000249[0] 1778278742.058370 trunc from 0x9e1903 to 0x8e5652 (0xfc2b0 bytes) 000251[0] 1778278742.088417 trunc from 0x8e5653 to 0x405f09 (0x4df749 bytes) 000252[0] 1778278742.100264 mapread 0x13b70d thru 0x13d647 (0x1f3b bytes) 000253[0] 1778278742.125724 trunc from 0x405f0a to 0x121d25 (0x2e41e4 bytes) 000254[1] 1778278742.155686 trunc from 0x121d26 to 0x292ae0 (0x170dbb bytes) 000255[1] 1778278742.193416 mapwrite 0x3f83f1 thru 0x3f90e9 (0xcf9 bytes) 000256[1] 1778278742.255851 trunc from 0x3f90ea to 0x8866e3 (0x48d5fa bytes) 000258[0] 1778278742.272537 write 0x092ee5 thru 0x093523 (0x63f bytes) 000260[1] 1778278742.304724 trunc from 0x8866e4 to 0x33b0eb (0x54b5f8 bytes) 000261[0] 1778278742.333230 mapread 0x0c1956 thru 0x0c2919 (0xfc4 bytes) 000262[0] 1778278742.355021 read 0x166286 thru 0x167e66 (0x1be1 bytes) 000266[0] 1778278742.398061 mapwrite 0x739ec1 thru 0x73b97c (0x1abc bytes) 000270[0] 1778278742.486588 mapread 0x35a652 thru 0x35c32b (0x1cda bytes) 000274[0] 1778278742.507207 mapread 0x1eefdc thru 0x1f0219 (0x123e bytes) 000275[0] 1778278742.516185 mapwrite 0x87e609 thru 0x87face (0x14c6 bytes) 000276[1] 1778278742.585491 mapread 0x8160c6 thru 0x817377 (0x12b2 bytes) 000280[1] 1778278742.662118 read 0x35727c thru 0x3582e5 (0x106a bytes) 000283[1] 1778278742.678433 mapread 0x191ba7 thru 0x1925cc (0xa26 bytes) truncating to largest ever: 0x968b6f 000285[0] 1778278742.694526 trunc from 0x87facf to 0x968b6e (0xe90a0 bytes) 000286[1] 1778278742.730986 mapread 0x3520e7 thru 0x35290b (0x825 bytes) 000289[0] 1778278742.795051 mapread 0x4a9957 thru 0x4a9eae (0x558 bytes) 000290[1] 1778278742.811232 mapwrite 0x17503a thru 0x175666 (0x62d bytes) 000291[1] 1778278742.894737 read 0x5831d9 thru 0x584044 (0xe6c bytes) 000292[1] 1778278742.914382 trunc from 0x968b6f to 0x551465 (0x417709 bytes) 000293[0] 1778278742.947244 mapwrite 0x91cf67 thru 0x91e6cc (0x1766 bytes) 000295[0] 1778278743.030688 write 0x60d82e thru 0x60eb8e (0x1361 bytes) 000297[1] 1778278743.048779 mapread 0x0cf0f2 thru 0x0d0d6a (0x1c79 bytes) 000298[1] 1778278743.080169 mapwrite 0x103f19 thru 0x104634 (0x71c bytes) 000299[0] 1778278743.137373 mapread 0x2c1203 thru 0x2c170b (0x509 bytes) 000300[0] 1778278743.155320 read 0x69c02e thru 0x69c0fc (0xcf bytes) 000301[0] 1778278743.164106 trunc from 0x91e6cd to 0x68d6ee (0x290fde bytes) 000304[1] 1778278743.199004 write 0x6dd6e5 thru 0x6debc7 (0x14e3 bytes) 000305[0] 1778278743.218901 read 0x23a632 thru 0x23b392 (0xd61 bytes) 000307[0] 1778278743.236060 mapwrite 0x34625a thru 0x347d94 (0x1b3b bytes) 000308[0] 1778278743.271381 mapread 0x217037 thru 0x2188ad (0x1877 bytes) 000309[0] 1778278743.285891 mapread 0x59d508 thru 0x59d9a7 (0x4a0 bytes) 000310[0] 1778278743.297917 trunc from 0x6debc8 to 0x26cd88 (0x471e3f bytes) 000311[1] 1778278743.310533 write 0x693e91 thru 0x6957c6 (0x1936 bytes) 000314[0] 1778278743.335164 mapwrite 0x79fa3d thru 0x79fcf8 (0x2bc bytes) 000315[1] 1778278743.390106 write 0x5dc29d thru 0x5dd2b2 (0x1016 bytes) 000317[0] 1778278743.442179 mapwrite 0x8eeb69 thru 0x8f09e5 (0x1e7d bytes) 000318[0] 1778278743.557639 mapwrite 0x5a1a26 thru 0x5a3571 (0x1b4c bytes) 000319[1] 1778278743.630014 mapwrite 0x33d2f3 thru 0x33f02c (0x1d3a bytes) 000322[1] 1778278743.707160 mapwrite 0x047ed2 thru 0x049dc6 (0x1ef5 bytes) 000323[0] 1778278743.745679 read 0x41de04 thru 0x41f9fc (0x1bf9 bytes) 000326[1] 1778278743.767561 read 0x85ee06 thru 0x85fce5 (0xee0 bytes) 000328[1] 1778278743.781314 mapread 0x363b1c thru 0x364829 (0xd0e bytes) 000329[0] 1778278743.792519 trunc from 0x8f09e6 to 0x228957 (0x6c808e bytes) 000330[1] 1778278743.805079 write 0x2e34b9 thru 0x2e3b65 (0x6ad bytes) 000331[1] 1778278743.816262 mapread 0x1f44d8 thru 0x1f61c8 (0x1cf1 bytes) 000332[1] 1778278743.833596 trunc from 0x2e3b66 to 0x8da672 (0x5f6b0d bytes) 000333[0] 1778278743.847328 write 0x49f911 thru 0x49fdd9 (0x4c9 bytes) 000334[0] 1778278743.861131 mapwrite 0x7f97d9 thru 0x7fb32c (0x1b54 bytes) 000335[0] 1778278743.901596 mapwrite 0x4a02b6 thru 0x4a0e29 (0xb74 bytes) 000339[0] 1778278743.950250 mapwrite 0x8b711a thru 0x8b76a6 (0x58d bytes) 000340[0] 1778278743.977579 read 0x879685 thru 0x87aa33 (0x13af bytes) 000344[0] 1778278743.993752 read 0x6dc732 thru 0x6dcb9b (0x46a bytes) 000345[0] 1778278743.995778 trunc from 0x8da673 to 0x41141e (0x4c9254 bytes) 000347[0] 1778278744.015980 mapread 0x33bee8 thru 0x33c254 (0x36d bytes) 000348[0] 1778278744.027043 mapread 0x208d80 thru 0x208ecd (0x14e bytes) 000350[1] 1778278744.030056 read 0x04e05e thru 0x04fde9 (0x1d8c bytes) 000351[0] 1778278744.038520 trunc from 0x41141f to 0x240e61 (0x1d05bd bytes) 000353[1] 1778278744.060837 trunc from 0x240e62 to 0x9292b7 (0x6e8456 bytes) 000355[0] 1778278744.077305 mapread 0x02a0d3 thru 0x02bea0 (0x1dce bytes) 000357[1] 1778278744.094757 write 0x5933c9 thru 0x593b3f (0x777 bytes) 000358[1] 1778278744.101054 read 0x25efaf thru 0x25fbf7 (0xc49 bytes) 000359[0] 1778278744.103567 write 0x0b69d6 thru 0x0b7430 (0xa5b bytes) 000363[1] 1778278744.132327 mapwrite 0x535531 thru 0x535db3 (0x883 bytes) 000367[1] 1778278744.174245 read 0x7da5d3 thru 0x7dbf06 (0x1934 bytes) 000368[1] 1778278744.180709 read 0x5c2a08 thru 0x5c3a04 (0xffd bytes) 000377[0] 1778278744.219367 write 0x89a2b0 thru 0x89a781 (0x4d2 bytes) 000378[0] 1778278744.226314 trunc from 0x9292b8 to 0x409909 (0x51f9ae bytes) 000379[0] 1778278744.235456 mapread 0x2a9692 thru 0x2ab2b4 (0x1c23 bytes) 000382[1] 1778278744.250113 mapwrite 0x16661f thru 0x16786b (0x124d bytes) 000383[0] 1778278744.286893 write 0x34c39c thru 0x34dd23 (0x1988 bytes) 000384[0] 1778278744.302358 mapread 0x2e19df thru 0x2e2f3e (0x1560 bytes) 000387[1] 1778278744.309932 write 0x91447f thru 0x915a62 (0x15e4 bytes) 000388[1] 1778278744.323473 mapread 0x84abc2 thru 0x84b49c (0x8db bytes) 000389[0] 1778278744.350217 trunc from 0x915a63 to 0x63cd02 (0x2d8d60 bytes) 000391[0] 1778278744.392719 mapread 0x2ed9b4 thru 0x2ee444 (0xa91 bytes) 000393[0] 1778278744.412483 write 0x3165c7 thru 0x317950 (0x138a bytes) 000397[1] 1778278744.425985 read 0x0b8577 thru 0x0b8a48 (0x4d2 bytes) 000400[1] 1778278744.450089 write 0x41fb3f thru 0x42086d (0xd2f bytes) 000401[1] 1778278744.461791 write 0x1517ed thru 0x152ea5 (0x16b9 bytes) 000402[0] 1778278744.469388 trunc from 0x63cd03 to 0x1dfa85 (0x45d27d bytes) 000403[0] 1778278744.499853 read 0x14de1b thru 0x14f83c (0x1a22 bytes) 000405[1] 1778278744.533718 mapwrite 0x045087 thru 0x045f32 (0xeac bytes) 000407[0] 1778278744.572348 mapwrite 0x25dd07 thru 0x25fbc4 (0x1ebe bytes) 000408[1] 1778278744.610316 read 0x0d3892 thru 0x0d4383 (0xaf2 bytes) 000411[1] 1778278744.646287 trunc from 0x25fbc5 to 0x30dcb4 (0xae0f0 bytes) 000412[0] 1778278744.662464 mapwrite 0x49fd8f thru 0x4a10e1 (0x1353 bytes) 000413[0] 1778278744.712462 mapread 0x1be8b4 thru 0x1bf8ec (0x1039 bytes) 000414[1] 1778278744.718429 write 0x61e7ae thru 0x61f311 (0xb64 bytes) 000415[0] 1778278744.728261 mapwrite 0x86391a thru 0x864feb (0x16d2 bytes) 000416[1] 1778278744.770430 mapread 0x3f6308 thru 0x3f6c36 (0x92f bytes) 000418[1] 1778278744.790384 read 0x347069 thru 0x348e64 (0x1dfc bytes) 000422[1] 1778278744.796799 trunc from 0x864fec to 0x359043 (0x50bfa8 bytes) 000423[1] 1778278744.815890 read 0x1850c1 thru 0x1850e3 (0x23 bytes) 000426[0] 1778278744.848280 read 0x1729ac thru 0x1740af (0x1704 bytes) 000427[0] 1778278744.884854 mapwrite 0x04e27d thru 0x04f6c4 (0x1448 bytes) truncating to largest ever: 0x9ef59d 000429[1] 1778278744.929943 trunc from 0x359044 to 0x9ef59c (0x696559 bytes) 000431[1] 1778278744.949003 trunc from 0x9ef59d to 0x0776f1 (0x977eab bytes) 000432[1] 1778278744.960092 write 0x7f172c thru 0x7f3121 (0x19f6 bytes) 000433[0] 1778278744.967364 read 0x41bf28 thru 0x41d16b (0x1244 bytes) 000434[0] 1778278744.977990 trunc from 0x7f3122 to 0x3a1b9d (0x451584 bytes) 000437[0] 1778278744.989424 trunc from 0x3a1b9e to 0x675e9b (0x2d42fe bytes) 000438[1] 1778278744.995372 trunc from 0x675e9c to 0x05eb9c (0x6172ff bytes) 000442[1] 1778278745.015064 write 0x07fbf5 thru 0x081757 (0x1b63 bytes) 000445[0] 1778278745.025191 read 0x01bbce thru 0x01ca15 (0xe48 bytes) 000447[0] 1778278745.039778 read 0x03c588 thru 0x03cd9e (0x817 bytes) 000451[0] 1778278745.049318 read 0x00ab37 thru 0x00ac7a (0x144 bytes) 000453[0] 1778278745.053864 read 0x0325f0 thru 0x0342a0 (0x1cb1 bytes) 000454[0] 1778278745.054357 mapwrite 0x565c6c thru 0x567beb (0x1f80 bytes) 000458[1] 1778278745.103112 mapwrite 0x2b0575 thru 0x2b0605 (0x91 bytes) 000462[0] 1778278745.148275 mapwrite 0x900a67 thru 0x90260d (0x1ba7 bytes) 000465[1] 1778278745.200554 trunc from 0x90260e to 0x638da4 (0x2c9869 bytes) 000468[1] 1778278745.221581 mapwrite 0x44a716 thru 0x44ba4c (0x1337 bytes) 000470[1] 1778278745.245450 read 0x3a43b0 thru 0x3a544b (0x109c bytes) 000471[0] 1778278745.247481 mapwrite 0x2311d0 thru 0x23220f (0x1040 bytes) 000472[0] 1778278745.269889 mapwrite 0x0479ef thru 0x047e43 (0x455 bytes) 000473[1] 1778278745.287938 read 0x22b369 thru 0x22cd56 (0x19ee bytes) 000475[0] 1778278745.296577 mapwrite 0x3a9d82 thru 0x3ab9b7 (0x1c36 bytes) 000476[0] 1778278745.322327 mapread 0x2158d4 thru 0x21680b (0xf38 bytes) 000478[1] 1778278745.335846 write 0x89b9ac thru 0x89d77a (0x1dcf bytes) 000483[0] 1778278745.349787 write 0x654d0e thru 0x65673e (0x1a31 bytes) 000484[1] 1778278745.358690 mapwrite 0x9dae36 thru 0x9dc36a (0x1535 bytes) 000485[0] 1778278745.393067 mapread 0x083b3b thru 0x084365 (0x82b bytes) 000486[0] 1778278745.403115 trunc from 0x9dc36b to 0x8cf29f (0x10d0cb bytes) 000487[0] 1778278745.415934 trunc from 0x8cf2a0 to 0x97b28a (0xabfeb bytes) 000488[0] 1778278745.420227 read 0x203ad4 thru 0x204269 (0x796 bytes) 000489[1] 1778278745.422289 mapwrite 0x410039 thru 0x4108f2 (0x8ba bytes) 000492[0] 1778278745.457545 trunc from 0x97b28b to 0x420359 (0x55af31 bytes) 000494[0] 1778278745.473140 mapread 0x31d070 thru 0x31dd04 (0xc95 bytes) 000495[1] 1778278745.479125 read 0x3c1ebd thru 0x3c2af6 (0xc3a bytes) 000497[0] 1778278745.487300 write 0x075be1 thru 0x076689 (0xaa9 bytes) 000498[0] 1778278745.496586 trunc from 0x42035a to 0x68fb10 (0x26f7b7 bytes) 000500[1] 1778278745.505150 trunc from 0x68fb11 to 0x9a53ff (0x3158ef bytes) All operations completed A-OK! PASS 16b (11s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 18:19:07 (1778278747) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 18:19:08 (1778278748) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.16207 s, 51.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.166799 s, 50.3 MB/s ldlm.namespaces.MGC192.168.204.157@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff934cc5473800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff934cd1bbc800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff934cc5473800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff934cc5473800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.31606 s, 31.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.96772 s, 108 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.91125 s, 115 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.96333 s, 53.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.85599 s, 36.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.32274 s, 79.3 MB/s PASS 16d (16s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 18:19:24 (1778278764) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.105215 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504494 s, 20.8 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 18:19:27 (1778278767) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 18:19:49 (1778278789) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 18:20:12 (1778278812) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 18:20:15 (1778278815) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 18:20:17 (1778278817) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.42824 s, 61.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.395742 s, 66.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.640915 s, 40.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.445252 s, 58.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.623573 s, 42.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.572484 s, 45.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.801834 s, 32.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.750168 s, 34.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.763773 s, 34.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.839645 s, 31.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.866603 s, 30.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.612387 s, 42.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.633648 s, 41.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.734139 s, 35.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.789816 s, 33.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.855078 s, 30.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.952054 s, 27.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.04987 s, 25.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.17032 s, 22.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (11s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 18:20:28 (1778278828) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Fri May 8 18:20:38 EDT 2026 + fsx 2 done: rc=0 Fri May 8 18:20:38 EDT 2026 + fsx 3 done: rc=0 Fri May 8 18:20:38 EDT 2026 + fsx 4 done: rc=0 Fri May 8 18:20:38 EDT 2026 + fsx 5 done: rc=0 Fri May 8 18:20:38 EDT 2026 PASS 16k (13s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 18:20:41 (1778278841) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 18:20:46 (1778278846) mmap test1: basic mmap operation (PASS, 0.036035s) mmap test2: MAP_PRIVATE not write back (PASS, 0.02645s) mmap test3: concurrent mmap ops on two nodes (PASS, 3.8811s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1236s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.035281s) mmap test6: check mmap write/read content on two nodes (PASS, 0.039173s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.43784s) mmap test11: multi-thread shared mmap (PASS, 0.46332s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0201s) PASS 18 (13s) == sanityn test 19: test concurrent uncached read races ========================================================================= 18:20:59 (1778278859) oleg457-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 18:21:00 (1778278860) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 18:21:03 (1778278863) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 18:21:05 (1778278865) now is 1778278865 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 new atime is 1778278926 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 18:22:08 (1778278928) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117297 401 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 18:22:11 (1778278931) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 18:22:13 (1778278933) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 18:22:16 (1778278936) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 18:22:17 (1778278937) utime: good mknod times 1778278936 <= 1778278937 <= 1778278937 for /mnt/lustre/f26a utime: good mknod times 1778278936 <= 1778278937 <= 1778278937 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 18:22:20 (1778278940) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 18:22:24 (1778278944) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-08 18:22:25.000000000 -0400 Birth: 2026-05-08 18:22:25.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-08 18:22:25.000000000 -0400 Birth: 2026-05-08 18:22:25.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 18:22:28 (1778278948) 0 PASS 30 (5s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 18:22:33 (1778278953) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 18:22:36 (1778278956) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.157@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 18:22:49 (1778278969) -rw-r--r-- 1 root root 0 May 8 18:22 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 18:22:55 (1778278975) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42854 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 18:22:58 (1778278978) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 18:23:02 (1778278982) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 18:23:04 (1778278984) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 18:23:05 (1778278985) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 18:23:07 (1778278987) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 18:23:08 (1778278988) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid in FULL state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid in FULL state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934cc5473800.ost_server_uuid in IDLE state after 0 sec oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff934cc5473800.ost_server_uuid in FULL state after 0 sec PASS 34 (93s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 18:24:41 (1778279081) fail_loc=0x80000317 Race attempt 0 Wait for 48900 49032 for 60 sec... - open/close 3672 (time 1778279093.66 total 10.00 last 367.19) total: 4000 open/close in 11.41 seconds: 350.47 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.157@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.157@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (65s) == sanityn test 36: handle ESTALE/open-unlink correctly == 18:25:46 (1778279146) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg457-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.444822 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.419361 s, 125 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.463799 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.36789 s, 143 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.425728 s, 123 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.431393 s, 122 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.432989 s, 121 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.408277 s, 128 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.445809 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.426641 s, 123 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.421713 s, 124 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7444480) after(7495680) PASS 36 (233s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 18:29:39 (1778279379) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 - create 5893 (time 1778279391.22 total 10.00 last 589.27) total: 10000 create in 16.63 seconds: 601.39 ops/second PASS 37 (21s) == sanityn test 39a: file mtime does not change after rename ========================================================== 18:30:00 (1778279400) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 18:30:02 (1778279402) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 18:30:06 (1778279406) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 18:30:09 (1778279409) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 18:30:11 (1778279411) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 18:30:20 (1778279420) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 18:30:28 (1778279428) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (8s) == sanityn test 40e: pdirops: rename and others ======================================================================== 18:30:37 (1778279437) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 18:30:44 (1778279444) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 18:30:50 (1778279450) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 18:30:55 (1778279455) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 18:31:00 (1778279460) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 18:31:05 (1778279465) Conflict PASS 41e (6s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 18:31:11 (1778279471) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 18:31:16 (1778279476) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 18:31:21 (1778279481) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 18:31:26 (1778279486) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (761s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 18:44:07 (1778280247) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 18:44:12 (1778280252) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 18:44:17 (1778280257) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 18:44:22 (1778280262) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 18:44:28 (1778280268) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 18:44:33 (1778280273) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 18:44:38 (1778280278) Conflict PASS 42g (4s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 18:44:43 (1778280283) Conflict PASS 42h (4s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 18:44:47 (1778280287) PASS 43a (14s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 18:45:01 (1778280301) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 18:45:06 (1778280306) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 18:45:11 (1778280311) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 18:45:16 (1778280316) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 18:45:21 (1778280321) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 18:45:26 (1778280326) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 18:45:31 (1778280331) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 18:45:36 (1778280336) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 18:45:37 (1778280337) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (42s) == sanityn test 43k: unlink vs create ==================== 18:46:19 (1778280379) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (408s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 18:53:07 (1778280787) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri May 8 18:53:11 EDT 2026 PASS 44a (5s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 18:53:12 (1778280792) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 18:53:18 (1778280798) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 18:53:23 (1778280803) Conflict PASS 44d (5s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 18:53:28 (1778280808) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 18:53:33 (1778280813) Conflict PASS 44f (5s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 18:53:38 (1778280818) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 18:53:44 (1778280824) Conflict PASS 44h (5s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 18:53:49 (1778280829) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 18:53:50 (1778280830) PASS 45a (28s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 18:54:18 (1778280858) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 18:54:23 (1778280863) Conflict PASS 45c (5s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 18:54:28 (1778280868) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 18:54:33 (1778280873) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 18:54:38 (1778280878) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (5s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 18:54:43 (1778280883) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 18:54:48 (1778280888) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 18:54:53 (1778280893) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 18:54:54 (1778280894) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (426s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 19:02:01 (1778281321) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 19:02:06 (1778281326) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 19:02:11 (1778281331) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 19:02:16 (1778281336) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 19:02:21 (1778281341) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 19:02:26 (1778281346) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 19:02:31 (1778281351) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 19:02:36 (1778281356) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 19:02:41 (1778281361) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 19:02:42 (1778281362) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 19:02:43 (1778281363) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 19:02:44 (1778281364) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 19:02:45 (1778281365) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 19:02:46 (1778281366) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 19:02:47 (1778281367) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 19:02:48 (1778281368) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 19:02:49 (1778281369) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.00452789 s, 2.3 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 19:02:56 (1778281376) 0+1 records in 0+1 records out 158 bytes copied, 0.00310473 s, 50.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (216935) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 19:03:00 (1778281380) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0028459 s, 360 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141632 s, 74.0 MB/s 3145728 PASS 51b (15s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 19:03:15 (1778281395) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00376444 s, 272 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 19:03:21 (1778281401) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122053 s, 85.9 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 19:03:24 (1778281404) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 19:03:28 (1778281408) llite.lustre-ffff934cc5473800.xattr_cache=1 llite.lustre-ffff934cd1bbc800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:29.000000000 -0400 Modify: 2026-05-08 19:03:29.000000000 -0400 Change: 2026-05-08 19:03:29.000000000 -0400 Birth: 2026-05-08 19:03:29.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:34.000000000 -0400 Modify: 2026-05-08 19:03:35.000000000 -0400 Change: 2026-05-08 19:03:35.000000000 -0400 Birth: 2026-05-08 19:03:35.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:40.000000000 -0400 Modify: 2026-05-08 19:03:40.000000000 -0400 Change: 2026-05-08 19:03:40.000000000 -0400 Birth: 2026-05-08 19:03:40.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:40.000000000 -0400 Modify: 2026-05-08 19:03:40.000000000 -0400 Change: 2026-05-08 19:03:40.000000000 -0400 Birth: 2026-05-08 19:03:40.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:45.000000000 -0400 Modify: 2026-05-08 19:03:45.000000000 -0400 Change: 2026-05-08 19:03:45.000000000 -0400 Birth: 2026-05-08 19:03:46.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:03:45.000000000 -0400 Modify: 2026-05-08 19:03:45.000000000 -0400 Change: 2026-05-08 19:03:45.000000000 -0400 Birth: 2026-05-08 19:03:46.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 19:03:52 (1778281432) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 19:04:00 (1778281440) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 19:04:07 (1778281447) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 19:04:20 (1778281460) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (9s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 19:04:29 (1778281469) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 19:04:30 (1778281470) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 19:04:43 (1778281483) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 19:04:57 (1778281497) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1778281498 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.90662 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (23s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 19:05:21 (1778281521) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1778281522 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.93788 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (36s) == sanityn test 60: Verify data_version behaviour ======== 19:05:56 (1778281556) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 19:05:58 (1778281558) PASS 70a (2s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 19:06:00 (1778281560) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 19:06:03 (1778281563) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00381796 s, 17.2 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 19:06:05 (1778281565) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00387247 s, 16.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 19:06:06 (1778281566) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 19:06:07 (1778281567) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00373031 s, 17.6 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 19:06:08 (1778281568) llite.lustre-ffff934cc5473800.xattr_cache=1 llite.lustre-ffff934cd1bbc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 19:06:10 (1778281570) llite.lustre-ffff934cc5473800.xattr_cache=1 llite.lustre-ffff934cd1bbc800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 19:06:12 (1778281572) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00262998 s, 389 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00232971 s, 440 kB/s 229702: get lock2 229705: get lock1 229702: done 229702: get lock1 229705: done 229705: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 229705: cannot get lock2: Resource deadlock avoided 229705: exit rc=0 fcntl cmd 7 failed: Resource deadlock avoided 229702: cannot get lock1: Resource deadlock avoided 229702: done 229702: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 19:06:18 (1778281578) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0184358 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102169 s, 103 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230285 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 19:06:25 (1778281585) opening files: ........ closing files: ........ PASS 76 (18s) == sanityn test 77a: check FIFO NRS policy =============== 19:06:43 (1778281603) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.128572 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296848 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486998 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717493 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714026 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0679883 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058255 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788821 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998819 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868699 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981317 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860208 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926724 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953499 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838629 s, 12.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672383 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920088 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163321 s, 64.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185004 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172319 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191646 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00647573 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267054 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311559 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0098612 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199385 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153828 s, 68.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288487 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221294 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216993 s, 48.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148253 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020294 s, 51.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129905 s, 80.7 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 19:06:45 (1778281605) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.118546 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362043 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359576 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300221 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779871 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782381 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900144 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984104 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109779 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913099 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893855 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105604 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113049 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110787 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120764 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121649 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131376 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123541 s, 84.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187307 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139285 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022529 s, 46.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422478 s, 248 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146561 s, 71.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347771 s, 30.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168841 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243093 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342787 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276047 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398018 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275805 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256228 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220274 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266204 s, 39.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.133765 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206615 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544893 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082441 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076742 s, 13.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723376 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885435 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891745 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738909 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661941 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631933 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0946736 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852592 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091001 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830757 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753238 s, 13.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0848497 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190441 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156099 s, 67.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199768 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151209 s, 69.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190443 s, 55.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209822 s, 50.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176734 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259635 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00904682 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248215 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128938 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156796 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113901 s, 92.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245741 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968413 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218574 s, 48.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 19:06:49 (1778281609) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158497 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368873 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374254 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369694 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493784 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059598 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0466497 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883975 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880277 s, 11.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10463 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881141 s, 11.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957375 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980842 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127355 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131975 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332753 s, 31.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251409 s, 41.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311676 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125471 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399345 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024272 s, 43.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306197 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141942 s, 7.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319406 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0389222 s, 26.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751489 s, 140 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288238 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230514 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112957 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312572 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246614 s, 42.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233723 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411153 s, 25.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.124024 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434259 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417666 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391662 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355841 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362318 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652591 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597722 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713651 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872729 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957678 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820129 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107232 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990688 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801982 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102725 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104962 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134632 s, 77.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190943 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232376 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147417 s, 71.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147719 s, 71.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222837 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184048 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425975 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119666 s, 87.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244449 s, 42.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198766 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00325982 s, 322 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240024 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00844405 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00776856 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826266 s, 127 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.114959 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211105 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328438 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544041 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598109 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073114 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535831 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734071 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521601 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798134 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749949 s, 14.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896561 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950006 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102957 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093129 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109163 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971912 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237313 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249486 s, 42.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181832 s, 57.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274194 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185856 s, 56.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235801 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277049 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305899 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237792 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370668 s, 28.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288074 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139626 s, 75.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165034 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153292 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164549 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165322 s, 63.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (5s) == sanityn test 77d: check TRR nrs policy ================ 19:06:54 (1778281614) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.110564 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195015 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046723 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396103 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046144 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559401 s, 18.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481606 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443453 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881156 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822237 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881842 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744181 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823787 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911437 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085079 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912489 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104589 s, 10.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133016 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125536 s, 83.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136902 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125461 s, 83.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216371 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017377 s, 60.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144847 s, 72.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837957 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182245 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019994 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235294 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556902 s, 188 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0049967 s, 210 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017811 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151302 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00197283 s, 532 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121153 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208162 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384177 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801464 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743169 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619791 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833657 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949706 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926594 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787329 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761471 s, 13.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950868 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914245 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725952 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736169 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732245 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900309 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146166 s, 71.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164035 s, 63.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160911 s, 65.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193043 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123215 s, 85.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01466 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225096 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176222 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889948 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137556 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177292 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629135 s, 167 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01226 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016303 s, 64.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626713 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368432 s, 285 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.109868 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204158 s, 51.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569553 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512627 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632412 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672983 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508536 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581484 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798802 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840106 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666431 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820829 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089292 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077585 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732729 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904216 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770187 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146038 s, 71.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184312 s, 56.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777502 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017448 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013653 s, 76.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159209 s, 65.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203172 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138381 s, 75.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234268 s, 44.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178283 s, 58.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263037 s, 39.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284966 s, 36.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134297 s, 78.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0045565 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00933716 s, 112 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126419 s, 82.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 19:06:58 (1778281618) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.57@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.119895 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223836 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465192 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489535 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481313 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640614 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0704209 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618214 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378795 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757238 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756638 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875836 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844159 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727224 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844832 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0729508 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792282 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141196 s, 74.3 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135376 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184525 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145982 s, 71.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174782 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359499 s, 292 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180354 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205987 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226099 s, 46.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00471308 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00777472 s, 135 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00685827 s, 153 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160384 s, 65.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151722 s, 69.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202942 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677792 s, 155 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121138 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015712 s, 66.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544678 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493221 s, 21.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587359 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488907 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558045 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061032 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728976 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659829 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904488 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807359 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830794 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816935 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800441 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831612 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745665 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00842617 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183775 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171639 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175959 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209882 s, 50.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015565 s, 67.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121753 s, 86.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190082 s, 55.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487697 s, 215 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723237 s, 145 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201204 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0141104 s, 74.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208171 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531053 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00497979 s, 211 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039055 s, 268 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.134172 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348227 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423628 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698728 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520599 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349939 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718549 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690101 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605678 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831321 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876247 s, 12.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124959 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879363 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920347 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149074 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267036 s, 39.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276081 s, 38.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031015 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206331 s, 50.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11943 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243611 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161971 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282625 s, 37.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240854 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238522 s, 44.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163676 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145725 s, 72.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439981 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304879 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113338 s, 92.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191898 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00698802 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167746 s, 62.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.115589 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201628 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473387 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692546 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589855 s, 17.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550821 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728016 s, 14.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642805 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832349 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436267 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642466 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673376 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503499 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996738 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122103 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889828 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938658 s, 11.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162105 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032858 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225521 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277588 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029405 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324061 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030321 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151143 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266748 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182011 s, 57.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378684 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294609 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201417 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224586 s, 46.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248211 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03082 s, 34.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (7s) == sanityn test 77f: check TBF JobID nrs policy ========== 19:07:05 (1778281625) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.116109 s, 144 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266709 s, 39.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734358 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637836 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065868 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079804 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619695 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860629 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823287 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786771 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074232 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716094 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926986 s, 11.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792189 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727297 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901618 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913919 s, 11.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01586 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115996 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128389 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122711 s, 85.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123096 s, 85.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150777 s, 69.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179188 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151646 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137981 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137282 s, 76.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200685 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00793756 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216515 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222176 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00351225 s, 299 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403643 s, 260 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171556 s, 97.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226648 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033962 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619189 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826345 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740275 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762095 s, 13.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621994 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824128 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829723 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101328 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911666 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108141 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099603 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994602 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869663 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939612 s, 11.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151984 s, 69.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143234 s, 73.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149997 s, 69.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172275 s, 60.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146261 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181423 s, 57.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238102 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179392 s, 58.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173957 s, 60.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019111 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00517639 s, 203 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114877 s, 91.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223484 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327298 s, 320 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00382659 s, 274 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256897 s, 40.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.115412 s, 145 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180329 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471253 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487003 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605985 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416766 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581395 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627159 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589117 s, 17.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766999 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091092 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912699 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790069 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938479 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103982 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111513 s, 9.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119122 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143673 s, 73.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177674 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188273 s, 55.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197467 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199899 s, 52.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196443 s, 53.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216819 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203126 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260822 s, 40.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242525 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024049 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204944 s, 51.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254035 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225213 s, 46.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228854 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260407 s, 40.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.120603 s, 139 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323844 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331626 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902734 s, 11.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff934cd1bbc800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812761 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882192 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783213 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762163 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882498 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953483 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897445 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874375 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846352 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916891 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868226 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919857 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907003 s, 11.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152645 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195994 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013628 s, 76.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182674 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162757 s, 64.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199974 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00823183 s, 127 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183448 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195866 s, 53.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180587 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200625 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0069833 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208134 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00417355 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491256 s, 213 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cd1bbc800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00172233 s, 609 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (7s) == sanityn test 77g: Change TBF type directly ============ 19:07:12 (1778281632) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 19:07:15 (1778281635) oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg457-client: oleg457-server: ssh exited with exit code 2 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 19:07:19 (1778281639) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (7s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 19:07:26 (1778281646) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121065 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176103 s, 59.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662175 s, 15.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648061 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069478 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746001 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728473 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698152 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820582 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815852 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866513 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828812 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712489 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762687 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653026 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590679 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700672 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210759 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194669 s, 53.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020675 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022135 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021308 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487302 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240936 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250989 s, 41.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962314 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226918 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179786 s, 58.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232715 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063588 s, 165 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563057 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263623 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012948 s, 81.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84405 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.3466 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 19:08:12 (1778281692) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21253 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247107 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523402 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423994 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450198 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049391 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342133 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413242 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621099 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162621 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162585 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172117 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178679 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171932 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193239 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201602 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176806 s, 5.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192271 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135244 s, 77.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225019 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289482 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252105 s, 41.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298748 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126946 s, 82.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191834 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627518 s, 167 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030217 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114247 s, 91.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252207 s, 41.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197649 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030451 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175114 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317098 s, 33.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7062 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9506 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21138 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751152 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646379 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637609 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809573 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525337 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727314 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768466 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067726 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163488 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163609 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180267 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165218 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153238 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154516 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186937 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168119 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157669 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017129 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144954 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183421 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147326 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128328 s, 81.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192071 s, 54.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198002 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219382 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145502 s, 72.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183507 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199739 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00388187 s, 270 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194969 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211417 s, 49.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0018033 s, 581 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7663 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9375 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (134s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 19:10:26 (1778281826) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21223 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523278 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658138 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508249 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650942 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555481 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593399 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598983 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471862 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163689 s, 6.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163733 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181078 s, 5.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164205 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16621 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173767 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161991 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16107 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018216 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196409 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140999 s, 74.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112531 s, 93.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211895 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915205 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192258 s, 54.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239669 s, 43.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023472 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219062 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037935 s, 276 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366315 s, 286 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218625 s, 48.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245009 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407506 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490383 s, 214 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7677 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9501 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.21232 s, 13.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796658 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085081 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777794 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736563 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707729 s, 14.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606468 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610945 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066043 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16544 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165512 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181622 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176327 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142802 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174654 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171406 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15733 s, 6.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164695 s, 63.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174676 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174976 s, 59.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197036 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181371 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170019 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225384 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170124 s, 61.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162119 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191032 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191384 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00372728 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017001 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0031457 s, 333 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225647 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00181649 s, 577 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7441 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9437 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (128s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 19:12:34 (1778281954) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.223073 s, 75.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266405 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443538 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252528 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559816 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539324 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622232 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532917 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563458 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487069 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636183 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796591 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791498 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907832 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877319 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825298 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103611 s, 10.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146207 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036254 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173671 s, 60.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135911 s, 77.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00962945 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165271 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184055 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225424 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216295 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265259 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181287 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269205 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231898 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206838 s, 50.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0037703 s, 278 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228873 s, 45.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83338 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.70821 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.57@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.57@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.227245 s, 73.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026812 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817629 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770061 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818918 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968605 s, 10.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976378 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090841 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888401 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858879 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971758 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938164 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06652 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240198 s, 4.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248586 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294872 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276286 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134514 s, 78.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131131 s, 80.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172187 s, 60.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258328 s, 40.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207437 s, 50.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244396 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193299 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00618772 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150133 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00450533 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228459 s, 45.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00808749 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100008 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273011 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274844 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018439 s, 56.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85541 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68786 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.57@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.225441 s, 74.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245335 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250743 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721913 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734982 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731832 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686175 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724894 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723171 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788269 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07407 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797305 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964388 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875823 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923192 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912303 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109056 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140331 s, 74.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164952 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154271 s, 68.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0112123 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172671 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066491 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181251 s, 57.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190874 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114186 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015953 s, 65.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404094 s, 259 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198901 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00381567 s, 275 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418109 s, 251 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306847 s, 342 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0036706 s, 286 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83064 s, 21.7 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.85999 s, 21.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.426446 s, 39.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240112 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567313 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467449 s, 22.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623356 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577552 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832301 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655569 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838486 s, 12.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725966 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774686 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863453 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811529 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217195 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217781 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242189 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256429 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122933 s, 85.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145087 s, 72.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155194 s, 67.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015072 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195865 s, 53.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131869 s, 79.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00390255 s, 269 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168602 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186807 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145913 s, 71.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180886 s, 58.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235209 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202397 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211864 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00441367 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00179342 s, 585 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.67124 s, 10.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71146 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83681 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.66044 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.825813 s, 20.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292907 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057431 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352943 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433991 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655257 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519118 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0593587 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552795 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159394 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174813 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184236 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192305 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20503 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187016 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203625 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192095 s, 5.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205484 s, 51.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198403 s, 52.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224383 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152235 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117839 s, 89.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170347 s, 61.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186743 s, 56.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113969 s, 92.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019058 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284584 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240969 s, 43.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00586041 s, 179 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251487 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058435 s, 179 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00755677 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00220953 s, 475 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5013 s, 5.4 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4096 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222817 s, 75.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022313 s, 47.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302982 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864387 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711189 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770942 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986391 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888021 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092283 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107025 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105619 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776004 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874909 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109322 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990057 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991181 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896576 s, 11.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113612 s, 92.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154637 s, 67.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181578 s, 57.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153825 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139953 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179468 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150206 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280534 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196636 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244808 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0039647 s, 264 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287194 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00681744 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.003401 s, 308 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00396013 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331859 s, 316 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.82775 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.70896 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.424527 s, 39.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286997 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612544 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086886 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816162 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10516 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897331 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924001 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983024 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118923 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911643 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984676 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111759 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281692 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288814 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322778 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318298 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199946 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014663 s, 71.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165445 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201044 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213814 s, 49.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018634 s, 56.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224296 s, 46.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217414 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298883 s, 35.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177782 s, 59.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170371 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204876 s, 51.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038689 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449936 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00462087 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430038 s, 244 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.64141 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.70877 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.62235 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.61226 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (345s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 19:18:19 (1778282299) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.223852 s, 74.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314231 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284517 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746013 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067416 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801437 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734301 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843973 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867619 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0895398 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113817 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091614 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102391 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883715 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108733 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983125 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941728 s, 11.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162575 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197084 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152257 s, 68.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157638 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165502 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019236 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769351 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00355688 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243818 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234784 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00639726 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202353 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00398876 s, 263 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409589 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00594081 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291949 s, 35.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.83679 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71008 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 19:18:56 (1778282336) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.57@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.57@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.222733 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262603 s, 39.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639282 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745024 s, 14.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820988 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524148 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743121 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799854 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103328 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794137 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845126 s, 12.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875141 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942699 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255857 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258912 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255903 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255086 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013537 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172659 s, 60.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208056 s, 50.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164133 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00412185 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165501 s, 63.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187202 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023679 s, 44.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017791 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009499 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180852 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00980753 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.006202 s, 169 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00747308 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00540671 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00241459 s, 434 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.82524 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63994 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (38s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 19:19:34 (1778282374) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.57@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.22645 s, 74.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289171 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546297 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276109 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580999 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050055 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670553 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625492 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547094 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881053 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818619 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865849 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764672 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939211 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11365 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117551 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109777 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201054 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234698 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195586 s, 53.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016034 s, 65.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232593 s, 45.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218137 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186522 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385255 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237123 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138577 s, 75.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274481 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253098 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235665 s, 44.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029697 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00397936 s, 264 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0059862 s, 175 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84269 s, 21.7 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93657 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (31s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 19:20:05 (1778282405) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.427147 s, 39.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216506 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456972 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058468 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363635 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382794 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534591 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605751 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508857 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462076 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831874 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858647 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105616 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270568 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297632 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390989 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37865 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104513 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180294 s, 58.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142285 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154464 s, 67.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399285 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149684 s, 70.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00627839 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428544 s, 24.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0171043 s, 61.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166939 s, 62.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00699584 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233474 s, 44.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00514055 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446156 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478245 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0498713 s, 21.0 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.6235 s, 10.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.70902 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.82457 s, 21.7 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.65612 s, 10.9 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (76s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 19:21:21 (1778282481) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.824121 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025365 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445246 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031869 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531706 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06037 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441008 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416825 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434048 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160269 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175055 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195982 s, 5.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190416 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18857 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21063 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206654 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182315 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187921 s, 55.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200649 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176926 s, 59.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207071 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193776 s, 54.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241824 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185103 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210564 s, 49.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190984 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160452 s, 65.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023934 s, 43.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00467556 s, 224 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402715 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00600801 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027476 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00527798 s, 199 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5199 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9455 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 19:22:27 (1778282547) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.26439 s, 63.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425682 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400712 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109872 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130963 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995884 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128599 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112923 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11145 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109271 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138072 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151547 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144137 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155199 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154423 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142305 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15422 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237118 s, 44.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269941 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192812 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293775 s, 35.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470741 s, 22.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321891 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350807 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361678 s, 29.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373092 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382948 s, 27.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381761 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481423 s, 21.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438625 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460845 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479577 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555667 s, 18.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88137 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71391 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.434399 s, 38.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280061 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590445 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313221 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551724 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783848 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626836 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0916602 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107423 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875876 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102962 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937476 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118219 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268145 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248657 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303113 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305125 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189786 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132102 s, 79.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230066 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158875 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229201 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147566 s, 71.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174962 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164379 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153464 s, 68.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176321 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203754 s, 51.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272486 s, 38.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019948 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261407 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258064 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270101 s, 38.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.72966 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.71546 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73568 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.69818 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (117s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 19:24:24 (1778282664) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre opts: Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg457-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg457-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg457-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg457-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg457-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63440/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.204.57,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg457-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg457-server: oleg457-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg457-client: oleg457-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Started clients oleg457-client.virtnet: 192.168.204.157@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff934cc88d6000.idle_timeout=debug osc.lustre-OST0000-osc-ffff934cdaf16800.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cc88d6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cdaf16800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234904 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246652 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283223 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751584 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873015 s, 12.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859929 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613239 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106658 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100147 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113898 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905057 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102532 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996443 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248174 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224365 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275545 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283063 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153282 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262528 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320109 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233686 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272379 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288537 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307316 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284837 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338532 s, 31.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337002 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351179 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345128 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362704 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371135 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371202 s, 28.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377436 s, 27.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.893 s, 21.4 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84589 s, 21.6 MB/s Read runtime is 5 s, speed is 20.000000 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.265138 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543233 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517729 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743352 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445617 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066287 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773942 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725502 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686002 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909126 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884528 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228724 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206003 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252599 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271943 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305953 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293394 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231143 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134759 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214889 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152176 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272481 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253669 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021695 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201459 s, 52.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271946 s, 38.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249394 s, 42.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257068 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025399 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256712 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342491 s, 30.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315728 s, 33.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308358 s, 34.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg457-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg457-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9048 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63919 s, 10.9 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (129s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 19:26:33 (1778282793) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (10s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 19:26:43 (1778282803) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 19:26:47 (1778282807) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg457-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.21434 s, 326 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg457-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0262 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (52s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 19:27:39 (1778282859) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.158657 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321319 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327144 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814015 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100374 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959638 s, 10.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887718 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cc88d6000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff934cc88d6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110485 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111358 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101432 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105518 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120102 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11392 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10431 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102493 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124424 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138511 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff934cc88d6000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff934cc88d6000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00837144 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302549 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365211 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00402231 s, 261 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161904 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182356 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383944 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00422885 s, 248 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315714 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00371813 s, 282 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249334 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420373 s, 249 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038539 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013146 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128327 s, 81.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271247 s, 38.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88548 s, 21.5 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.90577 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.199544 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329884 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029034 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113001 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101078 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850722 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113504 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127574 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12057 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133652 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138166 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139259 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113661 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139271 s, 7.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133043 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127412 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12659 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251505 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258328 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02479 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179168 s, 58.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155187 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248532 s, 42.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019442 s, 53.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223526 s, 46.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248325 s, 42.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233996 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245293 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253862 s, 41.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246251 s, 42.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258742 s, 40.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257544 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273729 s, 38.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg457-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg457-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.84885 s, 21.6 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91562 s, 21.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 19:28:45 (1778282925) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (6s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 19:28:51 (1778282931) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (46s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 19:29:37 (1778282977) mds.MDS.mdt.nrs_policies=tbf oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg457-client: oleg457-server: ssh exited with exit code 36 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (16s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 19:29:53 (1778282993) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 2 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269940 kill md_thread xargs PASS 77r (40s) == sanityn test 77s: Check TBF LRU shrinker ============== 19:30:33 (1778283033) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (23s) == sanityn test 77t: check TBF minrate parsing and output ========================================================== 19:30:56 (1778283056) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start fio jobid={fio.*} minrate=5 rate=20 realtime=1 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio rate=4: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio rate=4': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=10 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change fio minrate=21: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change fio minrate=21': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change fio rate=30 ost.OSS.ost_io.nrs_tbf_rule=change fio minrate=0 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start bad jobid={bad.*} minrate=31 rate=30: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start bad jobid={bad.*} minrate=31 rate=30': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop fio ost.OSS.ost_io.nrs_policies=fifo PASS 77t (12s) == sanityn test 77u: check TBF minrate on NID and generic rules ========================================================== 19:31:08 (1778283068) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start nid_min nid={0@lo 192.168.204.57@tcp} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=stop nid_min ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start combo_min jobid={dd.500}&opcode={ost_write} minrate=5 rate=20 ost.OSS.ost_io.nrs_tbf_rule=change combo_min minrate=0 ost.OSS.ost_io.nrs_tbf_rule=stop nid_min ost.OSS.ost_io.nrs_tbf_rule=stop combo_min ost.OSS.ost_io.nrs_policies=fifo PASS 77u (11s) == sanityn test 77v: verify minimum RPC rate under congestion ========================================================== 19:31:19 (1778283079) SKIP: sanityn test_77v need OSTs and MDTs on separate nodes SKIP 77v (1s) == sanityn test 77w: check TBF cost_model inference from rate format ========================================================== 19:31:20 (1778283080) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start cm_nid nid={0@lo 192.168.204.57@tcp} rate=2M ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=4M ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=8MiB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=500MB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid rate=1.5GiB/s ost.OSS.ost_io.nrs_tbf_rule=change cm_nid minrate=0 rate=5000 ost.OSS.ost_io.nrs_tbf_rule=start cm_both nid={0@lo} minrate=100K rate=1M oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start cm_bad nid={0@lo} rate=4G minrate=100: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start cm_bad nid={0@lo} rate=4G minrate=100': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start cm_bad2 nid={0@lo} rate=5000 minrate=1M: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start cm_bad2 nid={0@lo} rate=5000 minrate=1M': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=tbf nid oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start cm_mdt nid={0@lo} rate=1G: Invalid argument oleg457-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start cm_mdt nid={0@lo} rate=1G': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo ost.OSS.ost_io.nrs_tbf_rule=stop cm_both ost.OSS.ost_io.nrs_tbf_rule=stop cm_nid ost.OSS.ost_io.nrs_policies=fifo PASS 77w (16s) == sanityn test 77x: check TBF bandwidth suffix rate parsing ========================================================== 19:31:36 (1778283096) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start bw_nid nid={0@lo 192.168.204.57@tcp} rate=4G ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=512M ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=256MiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=1.5GiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=2G minrate=1G ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=4GiB/s minrate=512MiB/s ost.OSS.ost_io.nrs_tbf_rule=change bw_nid minrate=0 rate=5000 ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=5000 minrate=100 oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change bw_nid rate=1G: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change bw_nid rate=1G': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=change bw_nid rate=2G minrate=1G oleg457-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=change bw_nid rate=4X: Invalid argument oleg457-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='change bw_nid rate=4X': Invalid argument pdsh@oleg457-client: oleg457-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop bw_nid ost.OSS.ost_io.nrs_policies=fifo PASS 77x (15s) == sanityn test 77y: verify page-weighted cost_model under congestion ========================================================== 19:31:51 (1778283111) SKIP: sanityn test_77y need OSTs and MDTs on separate nodes SKIP 77y (1s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 19:31:52 (1778283112) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 19:31:56 (1778283116) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935966 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:31:57.000000000 -0400 Modify: 2026-05-08 19:31:57.000000000 -0400 Change: 2026-05-08 19:31:57.000000000 -0400 Birth: 2026-05-08 19:31:57.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (14s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 19:32:10 (1778283130) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 19:32:12 (1778283132) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 19:32:13 (1778283133) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 19:32:15 (1778283135) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 19:32:16 (1778283136) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 19:32:17 (1778283137) - open/close 1742 (time 1778283150.04 total 10.00 last 174.15) total: 2500 open/close in 13.16 seconds: 190.02 ops/second snapshot_time 1778283183.787484560 secs.nsecs start_time 1778283139.538064004 secs.nsecs elapsed_time 44.249420556 secs.nsecs open 2505 samples [usecs] 137 4871 2104194 2017026064 close 2504 samples [usecs] 81 2755 606305 184061339 mknod 2501 samples [usecs] 349 4762 1876916 1599437890 unlink 2500 samples [usecs] 495 34234 2834906 7791315872 mkdir 2 samples [usecs] 816 1146 1962 1979172 rmdir 2 samples [usecs] 2957 5210 8167 35887949 rename 2500 samples [usecs] 464 16494 6024976 21884454864 getattr 3761 samples [usecs] 6 3797 154289 68987665 parallel_rename_file 2500 samples [usecs] 464 16494 6024976 21884454864 crossdir_rename 2500 samples [usecs] 464 16494 6024976 21884454864 PASS 81d (48s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 19:33:05 (1778283185) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7568 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7568 PASS 82 (3s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 19:33:08 (1778283188) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:33:09 (1778283189) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 19:33:18 (1778283198) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1778283198 Finishing test test1 at 1778283199 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:33:22 (1778283202) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 19:33:23 (1778283203) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:33:25 (1778283205) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:33:26 (1778283206) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6099 0x17d3 0x240000400 1 6061 0x17ad 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6062 0x17ae 0x280000400 0 6100 0x17d4 0x240000400 rc1=0 and rc2=0 PASS 93 (10s) == sanityn test 94: signal vs CP callback race =========== 19:33:36 (1778283216) osc.lustre-OST0000-osc-ffff934cc88d6000.idle_timeout=0 osc.lustre-OST0000-osc-ffff934cdaf16800.idle_timeout=0 osc.lustre-OST0001-osc-ffff934cc88d6000.idle_timeout=0 osc.lustre-OST0001-osc-ffff934cdaf16800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0661968 s, 124 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.08841 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6845: 293213 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0249062 s, 164 kB/s osc.lustre-OST0000-osc-ffff934cc88d6000.idle_timeout=debug osc.lustre-OST0000-osc-ffff934cdaf16800.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cc88d6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff934cdaf16800.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:33:47 (1778283227) llite.lustre-ffff934cc88d6000.fast_read=0 llite.lustre-ffff934cdaf16800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0279631 s, 75.0 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff934cc88d6000.fast_read=1 llite.lustre-ffff934cdaf16800.fast_read=1 PASS 95a (15s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:34:02 (1778283242) llite.lustre-ffff934cc88d6000.fast_read=0 llite.lustre-ffff934cdaf16800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00323791 s, 3.8 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff934cc88d6000.fast_read=1 llite.lustre-ffff934cdaf16800.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:34:11 (1778283251) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:34:12 (1778283252) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00287658 s, 1.4 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:34:15 (1778283255) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0484757 s, 43.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (3s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:34:18 (1778283258) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0821097 s, 25.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 19:34:21 (1778283261) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00292437 s, 4.1 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 19:34:24 (1778283264) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00265477 s, 1.5 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 19:34:27 (1778283267) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0028074 s, 1.5 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 19:34:30 (1778283270) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00448109 s, 914 kB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 19:34:34 (1778283274) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1778283275 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1778283278 Finishing test test23 at 1778283278 PASS 103 (8s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:34:42 (1778283282) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 19:34:43 (1778283283) fail_loc=0x416 PASS 105 (24s) == sanityn test 106a: Verify the btime via statx() ======= 19:35:07 (1778283307) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:35:08 (1778283308) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00820714 s, 125 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938501 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:35:09.000000000 -0400 Modify: 2026-05-08 19:35:09.000000000 -0400 Change: 2026-05-08 19:35:09.000000000 -0400 Birth: 2026-05-08 19:35:09.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938501 -rw-r--r-- 1 0 0 1778283309 1778283309 1778283309 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938501 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:35:09.000000000 -0400 Modify: 2026-05-08 19:35:09.000000000 -0400 Change: 2026-05-08 19:35:09.000000000 -0400 Birth: 2026-05-08 19:35:09.000000000 -0400 1778283309 1024 1 PASS 106b (3s) == sanityn test 106c: Verify statx attributes mask ======= 19:35:11 (1778283311) PASS 106c (3s) == sanityn test 107a: Basic grouplock conflict =========== 19:35:14 (1778283314) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.106545 s, 98.4 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6108 0x17dc 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:35:19 (1778283319) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.11929 s, 87.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6069 0x17b5 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7568 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 19:35:28 (1778283328) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00454742 s, 2.2 MB/s llite.lustre-ffff934cc88d6000.hybrid_io=0 llite.lustre-ffff934cdaf16800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.26301 s, 2.0 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:35:35 (1778283335) 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre 192.168.204.157@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.157@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg457-client.virtnet /mnt/lustre (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg457-server' oleg457-server: oleg457-server.virtnet: executing load_modules_local oleg457-server: Loading modules from /home/green/git/lustre-release/lustre oleg457-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg457-server: detected 4 online CPUs by sysfs oleg457-server: MODOPTS_LIBCFS= oleg457-server: Force libcfs to create 2 CPU partitions Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre Starting client: oleg457-client.virtnet: -o user_xattr,flock 192.168.204.157@tcp:/lustre /mnt/lustre2 PASS 109 (40s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:36:15 (1778283375) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-08 19:36:16.000000000 -0400 Modify: 2026-05-08 19:36:16.000000000 -0400 Change: 2026-05-08 19:36:16.000000000 -0400 Birth: 2026-05-08 19:36:16.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg457-client.virtnet oleg457-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:39:01 (1778283541) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (2s) == sanityn test 112: update max-inherit in default LMV === 19:39:03 (1778283543) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (2s) == sanityn test 113: check servers of specified fs ======= 19:39:05 (1778283545) lustre-OST0000-osc-ffff934cc80f3800 active. lustre-OST0001-osc-ffff934cc80f3800 active. lustre-OST0000-osc-ffff934cc80f3800 active. lustre-OST0001-osc-ffff934cc80f3800 active. MGC192.168.204.157@tcp active. PASS 113 (4s) == sanityn test 114: implicit default LMV inherit ======== 19:39:09 (1778283549) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (2s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:39:11 (1778283551) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:39:12 (1778283552) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (2s) == sanityn test 121: trunc append race =================== 19:39:14 (1778283554) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00978819 s, 2.1 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0403261 s, 508 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (6s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:39:21 (1778283561) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1218 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1532 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1225 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1355 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1276 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1444 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1396 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1688 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1683 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2011 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1623 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1848 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.211 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2055 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3139 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.29 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3447 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2897 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg457-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff934cc80f3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff934cc80f3800.ost_server_uuid in FULL state after 0 sec PASS 200 (71s) cleanup: ====================================================== == sanityn test complete, duration 5301 sec ============== 19:40:33 (1778283633) === sanityn: start cleanup 19:40:33 (1778283633) === Stopping clients: oleg457-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg457-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:43:39 (1778283819) ===