-----============= acceptance-small: sanityn ============----- Wed May 20 16:56:15 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg335-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 16:56:28 (1779310588) === Starting client oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Mount client oleg335-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg335-client.virtnet: executing check_config_client /mnt/lustre oleg335-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg335-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff936b09417800.idle_timeout=debug osc.lustre-OST0000-osc-ffff936b107b5800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b09417800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b107b5800.idle_timeout=debug disable quota as required oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 16:56:49 (1779310609) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185979 s, 56.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7570] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 16:56:50 (1779310610) PASS 0 (5s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 16:56:55 (1779310615) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 16:57:00 (1779310620) -rw-r--r-- 1 root root 0 May 20 16:57 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (4s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 16:57:04 (1779310624) -rw-r--r-- 1 root root 0 May 20 16:57 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (4s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 16:57:08 (1779310628) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (4s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 16:57:12 (1779310632) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (4s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 16:57:16 (1779310636) total 1 -rwxrwxrwx 1 root root 0 May 20 16:57 f2a -rwxrwxrwx 1 root root 0 May 20 16:57 f2b total 1 -rwxrwxrwx 1 root root 0 May 20 16:57 f2a -rwxrwxrwx 1 root root 0 May 20 16:57 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0159552 s, 32.1 kB/s PASS 2e (4s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 16:57:21 (1779310641) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (2s) == sanityn test 2g: check blocks update on sync write ==== 16:57:23 (1779310643) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.266336 s, 7.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (5s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 16:57:28 (1779310648) PASS 3 (4s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 16:57:32 (1779310652) PASS 4 (5s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 16:57:37 (1779310657) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (4s) == sanityn test 6: remove of open file on other node ============================================================================ 16:57:42 (1779310662) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (4s) == sanityn test 7: remove of open directory on other node ======================================================================= 16:57:45 (1779310665) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (4s) == sanityn test 8: remove of open special file on other node ==================================================================== 16:57:50 (1779310670) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 16:57:54 (1779310674) PASS 9a (5s) == sanityn test 9b: append to striped sparse file ======== 16:57:59 (1779310679) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323542 s, 32.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0533178 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (5s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 16:58:04 (1779310684) 1+0 records in 1+0 records out 1 byte copied, 0.0211993 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0109385 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114067 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144324 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0081323 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184679 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0123457 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0091226 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128462 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00946504 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0108426 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010163 s, 0.1 kB/s PASS 10a (5s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 16:58:09 (1779310689) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0155288 s, 198 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0353715 s, 116 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000376076 s, 8.2 MB/s PASS 10b (4s) == sanityn test 11: execution of file opened for write should return error ============================================================== 16:58:14 (1779310694) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (4s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 16:58:18 (1779310698) warning: '-runtime' deprecated, use '-t runtime' instead - link 585 (time 1779310710.07 total 10.01 last 58.46) - link 1214 (time 1779310720.08 total 20.02 last 62.82) - link 1843 (time 1779310730.10 total 30.04 last 62.77) - link 2429 (time 1779310740.11 total 40.05 last 58.55) - link 3016 (time 1779310750.12 total 50.06 last 58.64) - link 3617 (time 1779310760.13 total 60.07 last 60.02) - link 4211 (time 1779310770.14 total 70.08 last 59.33) - link 4917 (time 1779310780.15 total 80.09 last 70.53) - link 5568 (time 1779310790.16 total 90.10 last 65.05) total: 6223 link in 100.01 seconds: 62.22 ops/second using seed 1417822789 running for 100 seconds - stat 7997 (time 1779310712 ; total 11 ; last 11) - stat 10000 (time 1779310714 ; total 13 ; last 2) - stat 18592 (time 1779310725 ; total 24 ; last 11) - stat 20000 (time 1779310726 ; total 25 ; last 1) - stat 28188 (time 1779310737 ; total 36 ; last 11) - stat 30000 (time 1779310739 ; total 38 ; last 2) - stat 38213 (time 1779310750 ; total 49 ; last 11) - stat 40000 (time 1779310753 ; total 52 ; last 3) - stat 43101 (time 1779310764 ; total 63 ; last 11) - stat 46745 (time 1779310775 ; total 74 ; last 11) - stat 50000 (time 1779310784 ; total 83 ; last 9) - stat 54045 (time 1779310795 ; total 94 ; last 11) total: 56435 stats in 100 seconds: 564.349976 stats/second - unlinked 0 (time 1779310751 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile6223) error: No such file or directory total: 6222 unlinks in 89 seconds: 69.910110 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21346) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21348) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 17:00:43 (1779310843) f13 f13 PASS 13 (4s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 17:00:47 (1779310847) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 22536 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (3s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 17:00:51 (1779310851) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 23115 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (3s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 17:00:54 (1779310854) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23692 Terminated $DIR1/$tdir/sleep 60 PASS 14b (3s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 17:00:57 (1779310857) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24275 Terminated $DIR1/$tdir/sleep 60 PASS 14c (4s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 17:01:01 (1779310861) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24871 Terminated $DIR1/$tdir/sleep 60 PASS 14d (5s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 17:01:06 (1779310866) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (36s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 17:01:43 (1779310903) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.181272 s, 46.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.137973 s, 60.8 MB/s Chance of close/open is 1 in 50 Seed set to 904 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1779310904.991916 mapwrite 0x7f93f3 thru 0x8074dc (0xe0ea bytes) 000005[1] 1779310905.113917 mapwrite 0x811c84 thru 0x81dd2f (0xc0ac bytes) truncating to largest ever: 0x5d61ba 000006[0] 1779310905.216428 trunc from 0x81dd30 to 0x5d61b9 (0x247b76 bytes) 000008[0] 1779310905.253820 trunc from 0x5d61ba to 0x2fa151 (0x2dc068 bytes) 000011[1] 1779310905.294613 mapread 0x074bfe thru 0x08235f (0xd762 bytes) 000014[0] 1779310905.330168 mapwrite 0x1db33a thru 0x1e49bb (0x9682 bytes) 000018[1] 1779310905.400843 mapread 0x1c00cb thru 0x1c7287 (0x71bd bytes) 000021[1] 1779310905.421561 mapread 0x0240f3 thru 0x0253fd (0x130b bytes) 000022[1] 1779310905.433114 mapread 0x2b6764 thru 0x2bacb0 (0x454d bytes) 000024[0] 1779310905.475990 mapread 0x0753a7 thru 0x077144 (0x1d9e bytes) 000026[0] 1779310905.508144 read 0x03605e thru 0x043213 (0xd1b6 bytes) 000028[0] 1779310905.549433 write 0x8435e0 thru 0x84a6d8 (0x70f9 bytes) 000029[0] 1779310905.560150 write 0x0a5935 thru 0x0ac3d9 (0x6aa5 bytes) 000032[1] 1779310905.577080 write 0x6f90e8 thru 0x6fac9c (0x1bb5 bytes) 000033[0] 1779310905.596021 mapread 0x0f0e07 thru 0x0f1edf (0x10d9 bytes) 000034[0] 1779310905.619211 mapwrite 0x2dfcb6 thru 0x2eec0b (0xef56 bytes) 000035[0] 1779310905.694997 mapread 0x0dbe94 thru 0x0df943 (0x3ab0 bytes) 000039[1] 1779310905.764552 read 0x10971a thru 0x10e4a4 (0x4d8b bytes) 000040[1] 1779310905.778898 write 0x85ff84 thru 0x862d40 (0x2dbd bytes) 000041[0] 1779310905.795220 mapwrite 0x998af6 thru 0x9a475b (0xbc66 bytes) 000042[0] 1779310905.900568 write 0x5165bd thru 0x51d7f7 (0x723b bytes) truncating to largest ever: 0x73bb70 000044[1] 1779310905.941273 trunc from 0x9a475c to 0x73bb6f (0x268bec bytes) 000045[1] 1779310905.975913 trunc from 0x73bb70 to 0x11b207 (0x620968 bytes) 000046[0] 1779310905.987071 write 0x7ddf16 thru 0x7ec847 (0xe932 bytes) 000047[0] 1779310906.011698 mapwrite 0x048a55 thru 0x054f7b (0xc527 bytes) 000048[0] 1779310906.097889 write 0x2de64a thru 0x2eb3f7 (0xcdae bytes) 000050[1] 1779310906.106468 mapread 0x7cbb4f thru 0x7ce551 (0x2a03 bytes) truncating to largest ever: 0x9957ce 000051[0] 1779310906.144534 trunc from 0x7ec848 to 0x9957cd (0x1a8f86 bytes) 000054[0] 1779310906.174364 read 0x571e27 thru 0x57480e (0x29e8 bytes) 000055[1] 1779310906.184950 read 0x352d75 thru 0x359941 (0x6bcd bytes) 000056[0] 1779310906.208959 mapwrite 0x090117 thru 0x0991d7 (0x90c1 bytes) 000059[1] 1779310906.302907 write 0x658d32 thru 0x667605 (0xe8d4 bytes) 000062[1] 1779310906.358610 read 0x434b84 thru 0x438008 (0x3485 bytes) 000072[1] 1779310906.494841 read 0x541bd8 thru 0x54e04b (0xc474 bytes) 000077[0] 1779310906.559666 mapread 0x95db45 thru 0x962634 (0x4af0 bytes) 000078[1] 1779310906.582350 read 0x94223d thru 0x94ae19 (0x8bdd bytes) 000082[1] 1779310906.620128 trunc from 0x9957ce to 0x384bb2 (0x610c1b bytes) 000083[0] 1779310906.639729 write 0x620b19 thru 0x62eb0e (0xdff6 bytes) 000084[1] 1779310906.657589 mapwrite 0x29e5e7 thru 0x2a6e19 (0x8833 bytes) 000085[1] 1779310906.710464 write 0x6b1b75 thru 0x6b4a3c (0x2ec8 bytes) 000088[1] 1779310906.724273 read 0x356696 thru 0x36574f (0xf0ba bytes) 000093[0] 1779310906.766142 trunc from 0x6b4a3d to 0x78ddbe (0xd9382 bytes) 000094[0] 1779310906.801189 mapread 0x2183d9 thru 0x22139a (0x8fc2 bytes) 000095[0] 1779310906.829794 mapwrite 0x5809cf thru 0x58e385 (0xd9b7 bytes) 000097[0] 1779310906.909941 read 0x2e6e49 thru 0x2e7a59 (0xc11 bytes) 000098[1] 1779310906.914165 write 0x8392bf thru 0x8450ed (0xbe2f bytes) 000099[0] 1779310906.938709 trunc from 0x8450ee to 0x78c1fd (0xb8ef0 bytes) 000103[0] 1779310906.999469 mapwrite 0x300a6e thru 0x308cdf (0x8272 bytes) 000105[1] 1779310907.060155 mapwrite 0x69925e thru 0x6a6fb9 (0xdd5c bytes) 000107[1] 1779310907.145133 read 0x401572 thru 0x409845 (0x82d4 bytes) 000110[1] 1779310907.190848 read 0x221794 thru 0x2224e5 (0xd52 bytes) 000114[0] 1779310907.229024 mapread 0x5abd56 thru 0x5adb76 (0x1e21 bytes) 000117[1] 1779310907.259046 write 0x421cad thru 0x4242f2 (0x2646 bytes) 000118[0] 1779310907.280487 mapread 0x3eae92 thru 0x3f9dae (0xef1d bytes) 000122[1] 1779310907.320969 mapread 0x63eb52 thru 0x6448e4 (0x5d93 bytes) 000125[1] 1779310907.351950 trunc from 0x78c1fe to 0x5baea2 (0x1d135b bytes) 000128[1] 1779310907.391776 trunc from 0x5baea3 to 0x7640f8 (0x1a9256 bytes) 000129[1] 1779310907.409932 write 0x192b5c thru 0x19d89d (0xad42 bytes) 000130[1] 1779310907.418753 read 0x5c2a01 thru 0x5c9020 (0x6620 bytes) 000131[0] 1779310907.433126 mapwrite 0x3dbb12 thru 0x3e5b3c (0xa02b bytes) 000133[0] 1779310907.508510 write 0x143243 thru 0x1526f4 (0xf4b2 bytes) 000134[0] 1779310907.520936 mapwrite 0x7d6f80 thru 0x7d95ef (0x2670 bytes) 000135[0] 1779310907.573409 trunc from 0x7d95f0 to 0x907dfc (0x12e80d bytes) 000136[0] 1779310907.583739 mapread 0x85eaeb thru 0x862694 (0x3baa bytes) 000137[1] 1779310907.589994 write 0x125518 thru 0x1342df (0xedc8 bytes) 000140[1] 1779310907.642241 read 0x0b9844 thru 0x0bca4c (0x3209 bytes) 000143[0] 1779310907.682134 mapread 0x784d8a thru 0x793e25 (0xf09c bytes) 000145[1] 1779310907.724677 mapread 0x39a488 thru 0x3a47e0 (0xa359 bytes) 000146[0] 1779310907.758940 trunc from 0x907dfd to 0x743330 (0x1c4acc bytes) 000148[1] 1779310907.799356 mapread 0x050189 thru 0x05f40e (0xf286 bytes) 000149[1] 1779310907.829928 mapwrite 0x314a3f thru 0x31d62e (0x8bf0 bytes) 000152[1] 1779310907.909463 trunc from 0x743331 to 0x6755a7 (0xcdd89 bytes) 000156[1] 1779310907.960452 mapwrite 0x933bdb thru 0x942fe1 (0xf407 bytes) 000158[0] 1779310908.033901 trunc from 0x942fe2 to 0x2bbbde (0x687403 bytes) 000159[1] 1779310908.056224 mapwrite 0x1cfcdb thru 0x1de809 (0xeb2f bytes) 000160[1] 1779310908.131787 read 0x0f5e47 thru 0x10178e (0xb948 bytes) 000161[0] 1779310908.167518 read 0x14c633 thru 0x15bd77 (0xf745 bytes) 000165[1] 1779310908.226361 trunc from 0x2bbbdf to 0x626c8b (0x36b0ad bytes) 000171[0] 1779310908.298972 write 0x33556c thru 0x3438ba (0xe34f bytes) 000173[0] 1779310908.341056 read 0x1ccda5 thru 0x1dc73a (0xf996 bytes) 000174[0] 1779310908.358377 mapread 0x48fe4a thru 0x49926c (0x9423 bytes) 000177[1] 1779310908.396986 read 0x5272b6 thru 0x52aec0 (0x3c0b bytes) 000178[1] 1779310908.419360 trunc from 0x626c8c to 0x149a61 (0x4dd22a bytes) 000180[0] 1779310908.471745 write 0x665f7e thru 0x66934e (0x33d1 bytes) 000183[0] 1779310908.509731 mapwrite 0x2c8dcf thru 0x2ca1d5 (0x1407 bytes) 000184[1] 1779310908.568110 read 0x0769ec thru 0x0842f3 (0xd908 bytes) 000185[0] 1779310908.580712 mapread 0x3a5795 thru 0x3a7e64 (0x26d0 bytes) 000187[1] 1779310908.593766 mapwrite 0x754bd3 thru 0x758bc7 (0x3ff5 bytes) 000188[1] 1779310908.667128 read 0x247441 thru 0x251bfb (0xa7bb bytes) 000191[0] 1779310908.703266 trunc from 0x758bc8 to 0x861cb7 (0x1090f0 bytes) 000195[0] 1779310908.751412 write 0x1a39f3 thru 0x1afd85 (0xc393 bytes) 000196[1] 1779310908.768413 mapread 0x72424b thru 0x731ce0 (0xda96 bytes) 000197[0] 1779310908.811422 mapwrite 0x967eb9 thru 0x96f245 (0x738d bytes) 000198[1] 1779310908.905814 read 0x79ede1 thru 0x7aacbb (0xbedb bytes) 000200[1] 1779310908.921921 mapread 0x87d8dd thru 0x886588 (0x8cac bytes) 000202[1] 1779310908.942027 mapread 0x175ded thru 0x183938 (0xdb4c bytes) 000205[0] 1779310908.958879 read 0x13cc22 thru 0x148ae2 (0xbec1 bytes) 000207[0] 1779310908.964755 read 0x15c588 thru 0x1646af (0x8128 bytes) 000208[0] 1779310908.969214 mapwrite 0x0f8bfc thru 0x10694e (0xdd53 bytes) 000210[0] 1779310909.048464 trunc from 0x96f246 to 0x949eca (0x2537b bytes) 000211[1] 1779310909.080709 mapwrite 0x054c2b thru 0x059b25 (0x4efb bytes) 000212[0] 1779310909.147220 read 0x4e08fc thru 0x4e6e64 (0x6569 bytes) 000214[0] 1779310909.177073 read 0x5ae382 thru 0x5b99e8 (0xb667 bytes) 000216[0] 1779310909.203632 trunc from 0x949ecb to 0x67c70f (0x2cd7bb bytes) 000220[0] 1779310909.239067 trunc from 0x67c710 to 0x1106f5 (0x56c01a bytes) 000225[0] 1779310909.288390 mapwrite 0x8699b0 thru 0x873b5d (0xa1ae bytes) 000228[1] 1779310909.368664 mapread 0x463590 thru 0x46f006 (0xba77 bytes) 000230[0] 1779310909.416513 write 0x332b3d thru 0x33a232 (0x76f6 bytes) 000232[0] 1779310909.422080 mapwrite 0x1d0764 thru 0x1dc78f (0xc02c bytes) 000233[1] 1779310909.471707 write 0x9a55da thru 0x9b1b50 (0xc577 bytes) 000234[0] 1779310909.479157 write 0x4b69f2 thru 0x4c4c6d (0xe27c bytes) 000237[0] 1779310909.532127 mapread 0x3d1d41 thru 0x3d7b51 (0x5e11 bytes) 000238[1] 1779310909.556437 write 0x9aef74 thru 0x9b6200 (0x728d bytes) 000239[1] 1779310909.580901 mapread 0x921e48 thru 0x92b325 (0x94de bytes) 000240[1] 1779310909.612239 read 0x5b7f52 thru 0x5bb69a (0x3749 bytes) 000241[1] 1779310909.633549 mapwrite 0x91f0e6 thru 0x9265eb (0x7506 bytes) 000242[1] 1779310909.673436 mapread 0x17e00c thru 0x1804fc (0x24f1 bytes) 000243[0] 1779310909.684179 mapwrite 0x287144 thru 0x28fd0a (0x8bc7 bytes) 000244[1] 1779310909.745179 mapwrite 0x748a24 thru 0x755581 (0xcb5e bytes) 000247[1] 1779310909.851613 trunc from 0x9b6201 to 0x47f5ec (0x536c14 bytes) 000248[1] 1779310909.876835 write 0x47b8ab thru 0x487141 (0xb897 bytes) 000251[1] 1779310909.897753 mapwrite 0x02ece2 thru 0x03af7a (0xc299 bytes) 000253[0] 1779310909.969095 mapwrite 0x629d7b thru 0x6399f3 (0xfc79 bytes) 000254[0] 1779310910.049548 mapwrite 0x8005de thru 0x804162 (0x3b85 bytes) 000258[0] 1779310910.128844 read 0x7520d4 thru 0x7613a7 (0xf2d4 bytes) 000259[0] 1779310910.146669 trunc from 0x804163 to 0x76b89e (0x988c4 bytes) truncating to largest ever: 0x9fbb23 000267[1] 1779310910.233961 trunc from 0x76b89f to 0x9fbb22 (0x290284 bytes) 000272[0] 1779310910.260567 mapread 0x34d08c thru 0x35126a (0x41df bytes) 000273[1] 1779310910.285621 mapwrite 0x22880b thru 0x22b133 (0x2929 bytes) 000274[0] 1779310910.321547 write 0x4836eb thru 0x48f560 (0xbe76 bytes) 000275[1] 1779310910.347037 read 0x620914 thru 0x62572e (0x4e1b bytes) 000277[1] 1779310910.388011 write 0x729400 thru 0x732432 (0x9033 bytes) 000282[0] 1779310910.420151 read 0x0af32a thru 0x0b492b (0x5602 bytes) 000284[1] 1779310910.476633 mapwrite 0x551ca4 thru 0x554b91 (0x2eee bytes) 000287[0] 1779310910.569925 mapwrite 0x6a4542 thru 0x6afac2 (0xb581 bytes) 000288[0] 1779310910.648798 mapwrite 0x35ded8 thru 0x36ab7e (0xcca7 bytes) 000291[1] 1779310910.710903 write 0x6acd5f thru 0x6bb7ab (0xea4d bytes) 000296[1] 1779310910.773156 write 0x415662 thru 0x41a1f8 (0x4b97 bytes) 000298[1] 1779310910.793601 write 0x5e8dab thru 0x5ec2b4 (0x350a bytes) 000302[0] 1779310910.842539 trunc from 0x9fbb23 to 0x6fed8d (0x2fcd95 bytes) 000304[1] 1779310910.866403 read 0x3cd245 thru 0x3daf3c (0xdcf8 bytes) 000305[0] 1779310910.880017 read 0x218cad thru 0x2228ab (0x9bff bytes) 000307[0] 1779310910.910458 write 0x7ed375 thru 0x7f0a52 (0x36de bytes) 000309[0] 1779310910.920137 mapwrite 0x2c3b54 thru 0x2c681d (0x2cca bytes) 000310[0] 1779310910.956651 mapread 0x0ac106 thru 0x0bbe69 (0xfd64 bytes) 000317[1] 1779310911.021525 write 0x939556 thru 0x94347a (0x9f25 bytes) 000319[0] 1779310911.054918 write 0x27dc44 thru 0x286304 (0x86c1 bytes) 000320[1] 1779310911.084537 write 0x5473ea thru 0x55240e (0xb025 bytes) 000322[1] 1779310911.119115 trunc from 0x94347b to 0x32f0f0 (0x61438a bytes) 000323[1] 1779310911.151336 mapwrite 0x76696d thru 0x76ab6d (0x4201 bytes) 000327[1] 1779310911.191129 mapread 0x0c01e4 thru 0x0c504c (0x4e69 bytes) 000330[1] 1779310911.208766 write 0x89c2cb thru 0x8a7509 (0xb23f bytes) 000333[1] 1779310911.211154 mapwrite 0x03d63f thru 0x04755a (0x9f1c bytes) 000334[0] 1779310911.257944 write 0x996caf thru 0x9a5f09 (0xf25b bytes) 000337[1] 1779310911.299080 write 0x42b746 thru 0x438bd8 (0xd493 bytes) 000340[1] 1779310911.357185 mapwrite 0x7bb55c thru 0x7cb003 (0xfaa8 bytes) 000341[0] 1779310911.585357 write 0x5bae87 thru 0x5c3587 (0x8701 bytes) 000342[1] 1779310911.593865 mapread 0x15af79 thru 0x16584c (0xa8d4 bytes) 000343[0] 1779310911.623869 mapwrite 0x9cd2a4 thru 0x9d147c (0x41d9 bytes) 000344[0] 1779310911.683641 write 0x827cf1 thru 0x83331a (0xb62a bytes) 000348[1] 1779310911.722680 write 0x1bf262 thru 0x1c1602 (0x23a1 bytes) 000354[0] 1779310911.783582 mapwrite 0x400586 thru 0x40966c (0x90e7 bytes) 000359[0] 1779310911.895622 mapwrite 0x997ad3 thru 0x99e7b5 (0x6ce3 bytes) 000360[1] 1779310911.945733 read 0x45c222 thru 0x4653f4 (0x91d3 bytes) 000362[1] 1779310911.957542 write 0x7c6208 thru 0x7d17c4 (0xb5bd bytes) 000363[0] 1779310911.974900 mapread 0x3c311a thru 0x3c90bb (0x5fa2 bytes) 000366[1] 1779310912.018429 trunc from 0x9d147d to 0x1ee0ad (0x7e33cf bytes) 000367[0] 1779310912.040222 mapwrite 0x43959d thru 0x4474b7 (0xdf1b bytes) 000368[1] 1779310912.123327 mapwrite 0x75b8ab thru 0x76208e (0x67e4 bytes) 000371[1] 1779310912.199118 mapread 0x5ee61a thru 0x5f586a (0x7251 bytes) 000372[1] 1779310912.214470 mapread 0x195bd4 thru 0x197d98 (0x21c5 bytes) 000374[0] 1779310912.237206 mapwrite 0x869646 thru 0x876a8b (0xd446 bytes) 000375[1] 1779310912.309850 mapwrite 0x2292af thru 0x2316c5 (0x8417 bytes) 000377[0] 1779310912.356701 mapwrite 0x8be95f thru 0x8c6a33 (0x80d5 bytes) 000379[0] 1779310912.398487 write 0x6c0383 thru 0x6cdf23 (0xdba1 bytes) 000380[1] 1779310912.423923 write 0x30d87f thru 0x31bd20 (0xe4a2 bytes) 000383[0] 1779310912.480619 write 0x59781c thru 0x59bdaa (0x458f bytes) 000384[0] 1779310912.505264 mapread 0x6fc7a0 thru 0x7086ed (0xbf4e bytes) 000387[0] 1779310912.556855 read 0x25d9ad thru 0x26ce4a (0xf49e bytes) 000390[0] 1779310912.602490 mapread 0x36e482 thru 0x37a84a (0xc3c9 bytes) 000393[0] 1779310912.657076 write 0x14c0ac thru 0x15093b (0x4890 bytes) 000394[0] 1779310912.684495 mapwrite 0x04dbf6 thru 0x054335 (0x6740 bytes) 000395[1] 1779310912.730104 trunc from 0x8c6a34 to 0x08ce47 (0x839bec bytes) 000397[1] 1779310912.762572 read 0x00d8a4 thru 0x01c461 (0xebbe bytes) 000398[1] 1779310912.770548 trunc from 0x08ce48 to 0x2c883a (0x23b9f3 bytes) 000404[0] 1779310912.808052 trunc from 0x2c883b to 0x7a700f (0x4de7d5 bytes) 000405[0] 1779310912.833225 mapwrite 0x9d99d8 thru 0x9e4696 (0xacbf bytes) 000410[1] 1779310912.928883 mapread 0x0871c8 thru 0x08f84a (0x8683 bytes) 000411[0] 1779310912.988847 mapread 0x8a1411 thru 0x8a689e (0x548e bytes) 000413[0] 1779310913.031445 trunc from 0x9e4697 to 0x6b2eb0 (0x3317e6 bytes) 000414[1] 1779310913.055258 write 0x59f525 thru 0x5a9ccd (0xa7a9 bytes) 000415[1] 1779310913.112415 write 0x624563 thru 0x631e28 (0xd8c6 bytes) 000416[1] 1779310913.131450 read 0x30ed1a thru 0x31b2af (0xc596 bytes) 000417[0] 1779310913.154987 write 0x12197f thru 0x123fff (0x2681 bytes) 000420[1] 1779310913.209928 write 0x1803e9 thru 0x183fd2 (0x3bea bytes) 000422[0] 1779310913.251720 mapread 0x2eb73a thru 0x2fb397 (0xfc5e bytes) 000428[1] 1779310913.313236 write 0x674a5c thru 0x683f5c (0xf501 bytes) 000429[0] 1779310913.332330 read 0x4947c0 thru 0x49cf97 (0x87d8 bytes) 000430[0] 1779310913.363198 read 0x690640 thru 0x697c17 (0x75d8 bytes) 000431[1] 1779310913.375488 mapread 0x65da81 thru 0x65f7f8 (0x1d78 bytes) 000433[1] 1779310913.393696 write 0x8103a6 thru 0x81fc02 (0xf85d bytes) 000440[0] 1779310913.438637 mapwrite 0x90c59f thru 0x90c8e0 (0x342 bytes) 000443[0] 1779310913.529753 write 0x44925d thru 0x44a350 (0x10f4 bytes) 000444[0] 1779310913.555427 write 0x0c2eb3 thru 0x0d1e1e (0xef6c bytes) 000445[0] 1779310913.565344 mapread 0x859e6f thru 0x8623fb (0x858d bytes) 000446[0] 1779310913.612064 mapread 0x680a16 thru 0x6874ec (0x6ad7 bytes) 000447[0] 1779310913.648704 read 0x7416f5 thru 0x748c04 (0x7510 bytes) 000448[0] 1779310913.673046 mapwrite 0x42624c thru 0x429ebd (0x3c72 bytes) 000449[0] 1779310913.732713 mapwrite 0x049ce6 thru 0x04d631 (0x394c bytes) 000450[1] 1779310913.808204 read 0x79dbe6 thru 0x7a4043 (0x645e bytes) 000453[0] 1779310913.878077 trunc from 0x90c8e1 to 0x0218c3 (0x8eb01d bytes) 000455[1] 1779310913.938698 write 0x65ef26 thru 0x66a778 (0xb853 bytes) 000456[1] 1779310913.970161 mapwrite 0x7f17ab thru 0x7f512a (0x3980 bytes) 000460[1] 1779310914.130663 trunc from 0x7f512b to 0x79fce7 (0x55443 bytes) 000463[0] 1779310914.197518 read 0x3c1698 thru 0x3d0f08 (0xf871 bytes) 000465[0] 1779310914.237493 write 0x0711fc thru 0x07ca47 (0xb84c bytes) 000469[0] 1779310914.311591 mapwrite 0x5268bc thru 0x52bca5 (0x53ea bytes) 000470[0] 1779310914.383181 trunc from 0x79fce8 to 0x195610 (0x60a6d7 bytes) 000471[1] 1779310914.399314 write 0x510162 thru 0x5178f3 (0x7792 bytes) 000472[0] 1779310914.434058 trunc from 0x5178f4 to 0x55cec8 (0x455d5 bytes) 000476[0] 1779310914.486340 mapwrite 0x71c882 thru 0x72239c (0x5b1b bytes) 000478[0] 1779310914.598811 trunc from 0x72239d to 0x30cc77 (0x415725 bytes) 000479[0] 1779310914.635392 mapread 0x200e18 thru 0x20694e (0x5b37 bytes) 000489[0] 1779310914.693682 mapwrite 0x14542a thru 0x14d67d (0x8254 bytes) 000490[0] 1779310914.736373 mapwrite 0x6e5db5 thru 0x6ed619 (0x7865 bytes) 000493[1] 1779310914.829026 mapwrite 0x2913d9 thru 0x29eb65 (0xd78d bytes) 000496[0] 1779310914.970834 write 0x155b10 thru 0x157481 (0x1972 bytes) 000497[0] 1779310914.997524 mapread 0x013b4e thru 0x01994b (0x5dfe bytes) 000498[1] 1779310915.030404 mapread 0x63499c thru 0x6435be (0xec23 bytes) 000499[0] 1779310915.094122 mapwrite 0x4dcedf thru 0x4e49ec (0x7b0e bytes) 000500[1] 1779310915.163395 mapwrite 0x41ad66 thru 0x4238f3 (0x8b8e bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 915 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000004[1] 1779310915.707224 write 0x3bc000 thru 0x3ca22a (0xe22b bytes) 000005[0] 1779310915.727770 read 0x303000 thru 0x309fab (0x6fac bytes) 000006[0] 1779310915.747908 read 0x08a000 thru 0x09630f (0xc310 bytes) 000008[1] 1779310915.787559 mapread 0x185000 thru 0x192ed3 (0xded4 bytes) 000010[1] 1779310915.806363 mapwrite 0x515000 thru 0x51c25b (0x725c bytes) 000014[0] 1779310915.928248 read 0x2e9000 thru 0x2f313b (0xa13c bytes) 000016[0] 1779310915.950804 read 0x15d000 thru 0x1658c4 (0x88c5 bytes) 000017[1] 1779310915.963915 read 0x3f7000 thru 0x3fa334 (0x3335 bytes) 000020[0] 1779310915.972101 write 0x123000 thru 0x129f2d (0x6f2e bytes) truncating to largest ever: 0x66c642 000025[0] 1779310916.003453 trunc from 0x51c25c to 0x66c641 (0x1503e6 bytes) 000026[1] 1779310916.027748 write 0x653000 thru 0x66245d (0xf45e bytes) 000027[0] 1779310916.053949 mapread 0x4b6000 thru 0x4ba422 (0x4423 bytes) 000028[1] 1779310916.102402 write 0x016000 thru 0x018bf8 (0x2bf9 bytes) 000029[1] 1779310916.116806 mapread 0x2b6000 thru 0x2bb18e (0x518f bytes) 000030[0] 1779310916.134214 read 0x10f000 thru 0x11320e (0x420f bytes) 000034[1] 1779310916.160181 trunc from 0x66c642 to 0x2213c9 (0x44b278 bytes) 000037[1] 1779310916.191845 write 0x563000 thru 0x56fc87 (0xcc88 bytes) 000039[1] 1779310916.219780 mapread 0x01c000 thru 0x0252d4 (0x92d5 bytes) 000041[1] 1779310916.254170 mapread 0x1af000 thru 0x1b2469 (0x346a bytes) truncating to largest ever: 0x733468 000042[0] 1779310916.289555 trunc from 0x56fc88 to 0x733467 (0x1c37e0 bytes) 000045[0] 1779310916.342345 trunc from 0x733468 to 0x716733 (0x1cd34 bytes) 000047[0] 1779310916.390682 mapread 0x446000 thru 0x4508a8 (0xa8a9 bytes) 000049[1] 1779310916.440175 write 0x551000 thru 0x5586d1 (0x76d2 bytes) 000050[1] 1779310916.464365 write 0x43e000 thru 0x44986d (0xb86e bytes) 000053[1] 1779310916.497911 trunc from 0x716734 to 0x27d710 (0x499023 bytes) 000056[1] 1779310916.559018 mapwrite 0x12a000 thru 0x1366d6 (0xc6d7 bytes) 000057[1] 1779310916.636459 mapwrite 0x54e000 thru 0x554df2 (0x6df3 bytes) 000058[1] 1779310916.736036 read 0x292000 thru 0x299346 (0x7347 bytes) 000060[1] 1779310916.750619 write 0x0ad000 thru 0x0b89d0 (0xb9d1 bytes) 000062[1] 1779310916.769096 mapwrite 0x19a000 thru 0x1a0e10 (0x6e11 bytes) 000064[1] 1779310916.861691 read 0x1f4000 thru 0x1ff396 (0xb397 bytes) 000065[1] 1779310916.872239 mapread 0x33c000 thru 0x341cde (0x5cdf bytes) 000071[0] 1779310916.940613 mapwrite 0x38b000 thru 0x392cf1 (0x7cf2 bytes) 000078[0] 1779310917.076647 mapwrite 0x311000 thru 0x31e0b6 (0xd0b7 bytes) 000079[1] 1779310917.146574 read 0x166000 thru 0x172048 (0xc049 bytes) 000081[1] 1779310917.179013 read 0x433000 thru 0x43ed08 (0xbd09 bytes) 000082[0] 1779310917.197232 read 0x053000 thru 0x058fd6 (0x5fd7 bytes) 000084[0] 1779310917.207125 write 0x95e000 thru 0x96897d (0xa97e bytes) 000085[1] 1779310917.247757 read 0x061000 thru 0x0681c9 (0x71ca bytes) 000087[1] 1779310917.269650 mapwrite 0x027000 thru 0x02b299 (0x429a bytes) 000089[0] 1779310917.332870 mapread 0x53f000 thru 0x54c6f1 (0xd6f2 bytes) 000091[1] 1779310917.381060 mapwrite 0x2f2000 thru 0x2fff86 (0xdf87 bytes) 000093[1] 1779310917.510081 write 0x679000 thru 0x684bf8 (0xbbf9 bytes) 000094[1] 1779310917.530784 mapread 0x475000 thru 0x476c22 (0x1c23 bytes) 000095[0] 1779310917.547389 write 0x0db000 thru 0x0e5307 (0xa308 bytes) 000097[1] 1779310917.609840 read 0x809000 thru 0x816b7f (0xdb80 bytes) 000099[0] 1779310917.641385 write 0x5e8000 thru 0x5ef443 (0x7444 bytes) 000102[1] 1779310917.672163 mapwrite 0x0d1000 thru 0x0d7183 (0x6184 bytes) 000104[0] 1779310917.791311 mapwrite 0x081000 thru 0x0901c2 (0xf1c3 bytes) 000105[1] 1779310917.909799 write 0x4d1000 thru 0x4dda39 (0xca3a bytes) 000107[0] 1779310917.945695 mapread 0x4b0000 thru 0x4b3fd7 (0x3fd8 bytes) 000111[0] 1779310917.991954 mapwrite 0x326000 thru 0x32f59a (0x959b bytes) 000112[1] 1779310918.088555 mapwrite 0x376000 thru 0x37744c (0x144d bytes) 000114[0] 1779310918.184255 write 0x21c000 thru 0x21c2f6 (0x2f7 bytes) 000115[0] 1779310918.218380 write 0x2ff000 thru 0x3095de (0xa5df bytes) 000116[1] 1779310918.237929 read 0x90d000 thru 0x90e71c (0x171d bytes) 000117[1] 1779310918.265748 mapwrite 0x781000 thru 0x78c253 (0xb254 bytes) 000118[0] 1779310918.371165 trunc from 0x96897e to 0x453d16 (0x514c67 bytes) 000119[0] 1779310918.420627 trunc from 0x453d17 to 0x7164ce (0x2c27b8 bytes) 000120[0] 1779310918.458153 write 0x2bb000 thru 0x2bd6e9 (0x26ea bytes) 000123[1] 1779310918.490081 read 0x11b000 thru 0x120251 (0x5252 bytes) 000124[0] 1779310918.516412 mapread 0x26b000 thru 0x27af76 (0xff77 bytes) 000127[1] 1779310918.538548 read 0x6f6000 thru 0x70339e (0xd39f bytes) 000129[0] 1779310918.550722 read 0x110000 thru 0x1182d0 (0x82d1 bytes) 000132[0] 1779310918.561573 trunc from 0x7164cf to 0x1e14b7 (0x535017 bytes) 000137[0] 1779310918.630135 mapwrite 0x578000 thru 0x582c8c (0xac8d bytes) 000139[0] 1779310918.781221 trunc from 0x582c8d to 0x2afa1f (0x2d326d bytes) 000140[1] 1779310918.826768 write 0x0a1000 thru 0x0a1cc1 (0xcc2 bytes) 000141[0] 1779310918.843108 read 0x107000 thru 0x109270 (0x2271 bytes) 000144[0] 1779310918.913455 mapread 0x196000 thru 0x19e58d (0x858e bytes) 000145[1] 1779310918.914745 read 0x0e4000 thru 0x0e50c0 (0x10c1 bytes) 000150[1] 1779310918.947262 mapread 0x297000 thru 0x2a1251 (0xa252 bytes) 000153[1] 1779310918.949062 write 0x979000 thru 0x9838fd (0xa8fe bytes) 000155[1] 1779310918.986939 read 0x167000 thru 0x16b9c3 (0x49c4 bytes) 000156[0] 1779310918.994243 mapwrite 0x21b000 thru 0x22294c (0x794d bytes) 000157[1] 1779310919.136921 mapwrite 0x04d000 thru 0x05bf8e (0xef8f bytes) 000160[1] 1779310919.302162 mapread 0x797000 thru 0x7a4cf4 (0xdcf5 bytes) 000163[0] 1779310919.371891 mapwrite 0x654000 thru 0x65f7a9 (0xb7aa bytes) 000164[1] 1779310919.474341 mapread 0x095000 thru 0x0a10a2 (0xc0a3 bytes) 000168[1] 1779310919.552894 read 0x431000 thru 0x4407dd (0xf7de bytes) 000170[0] 1779310919.562378 mapread 0x3d4000 thru 0x3df9e3 (0xb9e4 bytes) 000171[1] 1779310919.610581 mapread 0x71d000 thru 0x7260e8 (0x90e9 bytes) 000174[0] 1779310919.631174 mapwrite 0x16b000 thru 0x175aea (0xaaeb bytes) 000177[0] 1779310919.750771 mapwrite 0x818000 thru 0x820ef8 (0x8ef9 bytes) 000180[1] 1779310919.806848 mapwrite 0x507000 thru 0x50fb93 (0x8b94 bytes) 000182[0] 1779310919.881549 write 0x2b6000 thru 0x2c3302 (0xd303 bytes) 000183[0] 1779310919.906539 mapread 0x615000 thru 0x623cf0 (0xecf1 bytes) 000185[1] 1779310920.017046 write 0x56f000 thru 0x579040 (0xa041 bytes) 000191[0] 1779310920.088143 write 0x188000 thru 0x190e62 (0x8e63 bytes) 000193[1] 1779310920.132101 read 0x8fd000 thru 0x8fd95a (0x95b bytes) 000196[1] 1779310920.148940 mapread 0x4dd000 thru 0x4debb9 (0x1bba bytes) 000198[1] 1779310920.157755 trunc from 0x9838fe to 0x37e93d (0x604fc0 bytes) 000202[0] 1779310920.212671 read 0x10b000 thru 0x10b8b6 (0x8b7 bytes) 000207[0] 1779310920.258631 mapread 0x027000 thru 0x035388 (0xe389 bytes) 000212[0] 1779310920.260638 trunc from 0x37e93e to 0x5add41 (0x22f404 bytes) 000213[0] 1779310920.293156 trunc from 0x5add42 to 0x418b81 (0x1951c0 bytes) 000214[0] 1779310920.309833 read 0x169000 thru 0x178734 (0xf735 bytes) 000216[0] 1779310920.344190 write 0x59b000 thru 0x5aa365 (0xf366 bytes) 000217[0] 1779310920.361619 mapread 0x32f000 thru 0x33db6f (0xeb70 bytes) 000218[1] 1779310920.387760 mapwrite 0x641000 thru 0x64db84 (0xcb85 bytes) 000221[1] 1779310920.542084 read 0x095000 thru 0x09e8f3 (0x98f4 bytes) 000222[0] 1779310920.549788 write 0x20d000 thru 0x2151c2 (0x81c3 bytes) 000224[1] 1779310920.579022 mapread 0x050000 thru 0x051ec1 (0x1ec2 bytes) 000226[0] 1779310920.615645 trunc from 0x64db85 to 0x0702ac (0x5dd8d8 bytes) 000227[1] 1779310920.641571 mapread 0x00c000 thru 0x012b8c (0x6b8d bytes) 000229[1] 1779310920.677658 read 0x058000 thru 0x05a676 (0x2677 bytes) 000230[0] 1779310920.692752 trunc from 0x0702ad to 0x3a8c67 (0x3389bb bytes) 000232[0] 1779310920.756790 mapwrite 0x426000 thru 0x434aed (0xeaee bytes) 000239[0] 1779310920.932015 mapread 0x190000 thru 0x19df6c (0xdf6d bytes) 000240[1] 1779310920.967092 read 0x2fc000 thru 0x30b1b9 (0xf1ba bytes) 000242[0] 1779310920.984179 write 0x051000 thru 0x05570c (0x470d bytes) 000243[0] 1779310920.996506 trunc from 0x434aee to 0x48c889 (0x57d9c bytes) 000246[0] 1779310921.021925 mapwrite 0x70f000 thru 0x7124bc (0x34bd bytes) 000252[1] 1779310921.187714 mapread 0x56a000 thru 0x57044c (0x644d bytes) 000256[0] 1779310921.221543 write 0x093000 thru 0x09518e (0x218f bytes) 000258[0] 1779310921.264189 mapwrite 0x719000 thru 0x7216d9 (0x86da bytes) truncating to largest ever: 0x92682b 000260[1] 1779310921.352195 trunc from 0x7216da to 0x92682a (0x205151 bytes) truncating to largest ever: 0x9e9fbd 000261[0] 1779310921.393174 trunc from 0x92682b to 0x9e9fbc (0xc3792 bytes) 000263[0] 1779310921.447620 trunc from 0x9e9fbd to 0x55ff38 (0x48a084 bytes) 000265[0] 1779310921.486664 mapwrite 0x62a000 thru 0x62c1f2 (0x21f3 bytes) 000267[0] 1779310921.584184 write 0x871000 thru 0x880597 (0xf598 bytes) 000272[1] 1779310921.642953 mapwrite 0x3f1000 thru 0x3fa077 (0x9078 bytes) 000273[1] 1779310921.794056 write 0x82b000 thru 0x82bb4e (0xb4f bytes) 000275[0] 1779310921.801177 write 0x97f000 thru 0x984644 (0x5645 bytes) 000277[1] 1779310921.824583 trunc from 0x984645 to 0x01ad20 (0x969924 bytes) 000278[0] 1779310921.871811 mapread 0x00d000 thru 0x018539 (0xb53a bytes) 000279[0] 1779310921.921827 read 0x012000 thru 0x01485a (0x285b bytes) 000280[1] 1779310921.925160 read 0x011000 thru 0x01595f (0x4960 bytes) 000281[1] 1779310921.936149 mapread 0x016000 thru 0x01a84e (0x484f bytes) 000282[0] 1779310921.938068 trunc from 0x01ad21 to 0x4444ef (0x4297cf bytes) 000285[1] 1779310922.012586 mapread 0x3d8000 thru 0x3e3dd0 (0xbdd1 bytes) 000289[1] 1779310922.098085 mapread 0x0eb000 thru 0x0fadbb (0xfdbc bytes) 000291[1] 1779310922.099523 mapwrite 0x9f5000 thru 0x9f68b5 (0x18b6 bytes) 000293[1] 1779310922.309000 trunc from 0x9f68b6 to 0x5dc788 (0x41a12d bytes) 000294[1] 1779310922.363537 mapwrite 0x4dc000 thru 0x4e713d (0xb13e bytes) 000295[0] 1779310922.422939 read 0x4c7000 thru 0x4d2768 (0xb769 bytes) 000296[1] 1779310922.448030 write 0x769000 thru 0x777458 (0xe459 bytes) 000297[0] 1779310922.466248 read 0x046000 thru 0x04a956 (0x4957 bytes) 000299[1] 1779310922.506525 mapwrite 0x4a1000 thru 0x4a113e (0x13f bytes) 000300[1] 1779310922.580394 write 0x2c4000 thru 0x2c55a6 (0x15a7 bytes) 000302[1] 1779310922.632910 mapread 0x640000 thru 0x641b0f (0x1b10 bytes) 000306[1] 1779310922.691473 mapwrite 0x7a4000 thru 0x7b10f2 (0xd0f3 bytes) 000309[1] 1779310922.887123 trunc from 0x7b10f3 to 0x3ef344 (0x3c1dae bytes) 000312[1] 1779310922.948902 read 0x3d3000 thru 0x3e1f5d (0xef5e bytes) 000313[0] 1779310923.010933 read 0x1a8000 thru 0x1b0080 (0x8081 bytes) 000315[0] 1779310923.067575 trunc from 0x3ef345 to 0x163e1e (0x28b526 bytes) 000317[1] 1779310923.116165 read 0x037000 thru 0x03d024 (0x6025 bytes) 000319[0] 1779310923.152596 write 0x3bc000 thru 0x3be9b8 (0x29b9 bytes) 000320[1] 1779310923.169515 trunc from 0x3be9b9 to 0x1925ed (0x22c3cb bytes) 000323[0] 1779310923.203595 mapread 0x05d000 thru 0x067338 (0xa339 bytes) 000324[0] 1779310923.276123 trunc from 0x1925ee to 0x17e58e (0x1405f bytes) 000326[1] 1779310923.327114 mapread 00000000 thru 0x00d353 (0xd354 bytes) 000327[1] 1779310923.460707 read 0x02a000 thru 0x035391 (0xb392 bytes) 000328[0] 1779310923.483157 write 0x62d000 thru 0x63a023 (0xd024 bytes) 000331[1] 1779310923.523108 write 0x3a2000 thru 0x3ae82f (0xc830 bytes) 000333[1] 1779310923.612318 read 0x179000 thru 0x17e00b (0x500c bytes) 000335[0] 1779310923.664409 write 0x5f0000 thru 0x5f38b4 (0x38b5 bytes) 000337[0] 1779310923.739677 mapread 0x099000 thru 0x0a7eae (0xeeaf bytes) 000338[0] 1779310923.844203 mapread 0x4db000 thru 0x4e3880 (0x8881 bytes) 000339[1] 1779310923.859216 mapread 0x252000 thru 0x25c206 (0xa207 bytes) 000340[0] 1779310923.943734 write 0x25a000 thru 0x26776c (0xd76d bytes) 000343[1] 1779310924.007199 write 0x82d000 thru 0x83839f (0xb3a0 bytes) 000346[1] 1779310924.060822 mapwrite 0x60e000 thru 0x614718 (0x6719 bytes) 000347[0] 1779310924.132217 write 0x638000 thru 0x63f5b9 (0x75ba bytes) 000348[1] 1779310924.170880 mapwrite 0x193000 thru 0x1976bd (0x46be bytes) 000349[1] 1779310924.249250 read 0x483000 thru 0x491754 (0xe755 bytes) 000350[1] 1779310924.290274 write 0x54d000 thru 0x551ef1 (0x4ef2 bytes) 000351[0] 1779310924.298160 mapread 0x3b2000 thru 0x3be63e (0xc63f bytes) 000355[1] 1779310924.396147 write 0x04c000 thru 0x04f48a (0x348b bytes) 000357[1] 1779310924.429225 trunc from 0x8383a0 to 0x35dbe8 (0x4da7b7 bytes) 000360[1] 1779310924.491370 mapread 0x246000 thru 0x2485be (0x25bf bytes) 000362[1] 1779310924.533237 mapwrite 0x995000 thru 0x99b383 (0x6384 bytes) 000364[0] 1779310924.644270 trunc from 0x99b384 to 0x692e82 (0x308501 bytes) 000365[1] 1779310924.669452 write 0x044000 thru 0x045d38 (0x1d39 bytes) 000371[1] 1779310924.725761 trunc from 0x692e83 to 0x0211d1 (0x671cb1 bytes) 000373[1] 1779310924.746028 mapread 0x015000 thru 0x020658 (0xb659 bytes) 000374[1] 1779310924.769624 write 0x20c000 thru 0x20de1f (0x1e20 bytes) 000375[1] 1779310924.790030 mapread 0x029000 thru 0x030295 (0x7296 bytes) 000380[0] 1779310924.831606 write 0x69f000 thru 0x6ab335 (0xc336 bytes) 000381[1] 1779310924.851688 write 0x418000 thru 0x42049b (0x849c bytes) 000382[0] 1779310924.874587 read 0x18c000 thru 0x19b7bc (0xf7bd bytes) 000383[0] 1779310924.904112 mapread 0x3e9000 thru 0x3f2a0d (0x9a0e bytes) 000384[1] 1779310924.929720 mapread 0x1f8000 thru 0x207de9 (0xfdea bytes) 000385[1] 1779310924.975130 trunc from 0x6ab336 to 0x5a94c8 (0x101e6d bytes) 000389[0] 1779310925.057629 mapwrite 0x85f000 thru 0x86c445 (0xd446 bytes) 000390[1] 1779310925.190563 mapwrite 0x944000 thru 0x949ad9 (0x5ada bytes) 000396[0] 1779310925.283735 read 0x712000 thru 0x715fe6 (0x3fe7 bytes) 000399[0] 1779310925.304242 write 0x0f6000 thru 0x0ff95c (0x995d bytes) 000402[0] 1779310925.340225 trunc from 0x949ada to 0x273707 (0x6d63d2 bytes) 000403[1] 1779310925.379116 trunc from 0x273708 to 0x0da8a3 (0x198e64 bytes) 000406[0] 1779310925.436868 mapwrite 0x72d000 thru 0x737307 (0xa308 bytes) 000407[1] 1779310925.540162 write 0x2ad000 thru 0x2b3d5f (0x6d60 bytes) 000408[1] 1779310925.554819 read 0x545000 thru 0x551be9 (0xcbea bytes) 000409[1] 1779310925.580892 trunc from 0x737308 to 0x8ea82d (0x1b3526 bytes) 000410[1] 1779310925.607463 read 0x571000 thru 0x57ca1e (0xba1f bytes) 000411[1] 1779310925.624713 mapwrite 0x259000 thru 0x264706 (0xb707 bytes) 000412[0] 1779310925.722008 trunc from 0x8ea82e to 0x8baa51 (0x2fddc bytes) 000413[1] 1779310925.767529 read 0x4cd000 thru 0x4d9304 (0xc305 bytes) 000416[0] 1779310925.791606 trunc from 0x8baa52 to 0x98349c (0xc8a4b bytes) 000417[1] 1779310925.818213 write 0x600000 thru 0x605a49 (0x5a4a bytes) 000419[1] 1779310925.867062 write 0x1a6000 thru 0x1b567b (0xf67c bytes) 000420[1] 1779310925.874100 read 0x30a000 thru 0x30a7df (0x7e0 bytes) 000421[1] 1779310925.878831 mapread 0x028000 thru 0x02d360 (0x5361 bytes) 000424[0] 1779310925.899961 write 0x399000 thru 0x39b971 (0x2972 bytes) 000428[1] 1779310925.949181 mapread 0x414000 thru 0x4207ac (0xc7ad bytes) 000429[0] 1779310925.987190 mapwrite 0x529000 thru 0x535b0d (0xcb0e bytes) 000430[0] 1779310926.090412 write 0x5e3000 thru 0x5e6258 (0x3259 bytes) 000431[1] 1779310926.120123 mapread 0x761000 thru 0x767631 (0x6632 bytes) 000432[0] 1779310926.178923 mapread 0x482000 thru 0x48fc23 (0xdc24 bytes) 000434[0] 1779310926.222025 read 0x5e2000 thru 0x5eeb4f (0xcb50 bytes) 000436[0] 1779310926.236380 read 0x2cb000 thru 0x2d1916 (0x6917 bytes) 000437[1] 1779310926.249233 trunc from 0x98349d to 0x370280 (0x61321c bytes) 000438[0] 1779310926.299195 mapread 0x334000 thru 0x33b9be (0x79bf bytes) 000440[0] 1779310926.344006 write 0x4b2000 thru 0x4ba387 (0x8388 bytes) 000441[1] 1779310926.375044 mapwrite 0x910000 thru 0x91b1a8 (0xb1a9 bytes) 000442[1] 1779310926.489232 write 0x2bc000 thru 0x2bfd7e (0x3d7f bytes) 000444[0] 1779310926.526046 mapread 0x522000 thru 0x5225fb (0x5fc bytes) 000445[0] 1779310926.557264 mapread 0x596000 thru 0x5a56ec (0xf6ed bytes) 000446[0] 1779310926.582691 trunc from 0x91b1a9 to 0x7e2245 (0x138f63 bytes) 000447[0] 1779310926.609152 write 0x39b000 thru 0x3a2aee (0x7aef bytes) 000448[1] 1779310926.614835 mapread 0x27f000 thru 0x285fcd (0x6fce bytes) 000450[1] 1779310926.655841 write 0x553000 thru 0x561af2 (0xeaf3 bytes) 000452[0] 1779310926.703848 mapwrite 0x44e000 thru 0x45cc93 (0xec94 bytes) 000453[1] 1779310926.825177 read 0x4a6000 thru 0x4ad008 (0x7009 bytes) 000454[1] 1779310926.853730 mapwrite 0x7b8000 thru 0x7c2400 (0xa401 bytes) 000459[1] 1779310926.944242 mapread 0x692000 thru 0x69eef8 (0xcef9 bytes) 000460[1] 1779310926.987271 trunc from 0x7e2246 to 0x00550a (0x7dcd3b bytes) 000462[0] 1779310927.023875 mapwrite 0x5bf000 thru 0x5caed9 (0xbeda bytes) 000464[1] 1779310927.148179 trunc from 0x5caeda to 0x2ca413 (0x300ac6 bytes) 000466[1] 1779310927.167785 mapread 0x193000 thru 0x19a1a5 (0x71a6 bytes) 000468[0] 1779310927.198809 read 0x10a000 thru 0x116aa9 (0xcaaa bytes) 000470[1] 1779310927.232424 trunc from 0x2ca414 to 0x74bc6d (0x48185a bytes) 000471[0] 1779310927.278613 trunc from 0x74bc6e to 0x9db64f (0x28f9e2 bytes) 000473[1] 1779310927.332364 read 0x62f000 thru 0x63e8f8 (0xf8f9 bytes) 000474[0] 1779310927.353218 mapwrite 0x2ec000 thru 0x2fb0c6 (0xf0c7 bytes) 000477[0] 1779310927.460039 mapwrite 0x7df000 thru 0x7e8c07 (0x9c08 bytes) 000478[0] 1779310927.526153 read 0x06e000 thru 0x078b8f (0xab90 bytes) 000479[1] 1779310927.535830 write 0x601000 thru 0x605dde (0x4ddf bytes) 000481[1] 1779310927.575042 write 0x643000 thru 0x646302 (0x3303 bytes) 000483[1] 1779310927.580925 mapread 0x631000 thru 0x63f4b0 (0xe4b1 bytes) 000484[1] 1779310927.597839 mapread 0x704000 thru 0x70fbff (0xbc00 bytes) 000486[1] 1779310927.643760 trunc from 0x9db650 to 0x4aaa8f (0x530bc0 bytes) 000488[0] 1779310927.672611 write 0x137000 thru 0x140054 (0x9055 bytes) 000492[0] 1779310927.738083 write 0x8d6000 thru 0x8e17a9 (0xb7aa bytes) 000496[0] 1779310927.781018 write 0x4b8000 thru 0x4bfb60 (0x7b61 bytes) All operations completed A-OK! PASS 16a (27s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 17:02:10 (1779310930) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.245287 s, 34.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.195948 s, 42.8 MB/s Chance of close/open is 1 in 50 Seed set to 932 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1779310932.819822 mapwrite 0x7ea4a7 thru 0x7eba19 (0x1573 bytes) 000007[0] 1779310932.918410 mapread 0x72032c thru 0x721785 (0x145a bytes) 000009[0] 1779310932.939619 read 0x0fae3c thru 0x0fbd38 (0xefd bytes) 000012[0] 1779310932.958477 read 0x73e0da thru 0x73e8f4 (0x81b bytes) 000013[1] 1779310932.962663 write 0x4612ef thru 0x4626ab (0x13bd bytes) 000015[0] 1779310932.977585 read 0x7eb210 thru 0x7eba19 (0x80a bytes) 000018[1] 1779310932.999149 mapwrite 0x3fb512 thru 0x3fb9d4 (0x4c3 bytes) 000020[1] 1779310933.050157 mapwrite 0x20266d thru 0x203aff (0x1493 bytes) 000022[0] 1779310933.115263 mapwrite 0x80c4c4 thru 0x80d993 (0x14d0 bytes) 000023[0] 1779310933.208767 write 0x6c5af9 thru 0x6c7486 (0x198e bytes) 000027[0] 1779310933.246814 mapwrite 0x3872d7 thru 0x387ebf (0xbe9 bytes) 000028[1] 1779310933.290882 write 0x922a03 thru 0x924575 (0x1b73 bytes) 000029[1] 1779310933.308005 mapwrite 0x5e4479 thru 0x5e5970 (0x14f8 bytes) 000033[0] 1779310933.383662 mapread 0x49fb9f thru 0x4a0efa (0x135c bytes) 000034[0] 1779310933.408150 read 0x656a9c thru 0x65832d (0x1892 bytes) 000036[1] 1779310933.433014 write 0x3b6599 thru 0x3b6f31 (0x999 bytes) 000038[1] 1779310933.455016 read 0x7a25df thru 0x7a288c (0x2ae bytes) truncating to largest ever: 0x35e81b 000040[1] 1779310933.473179 trunc from 0x924576 to 0x35e81a (0x5c5d5b bytes) truncating to largest ever: 0x47b94f 000041[0] 1779310933.503664 trunc from 0x35e81b to 0x47b94e (0x11d134 bytes) 000043[0] 1779310933.548184 mapwrite 0x3c1ee0 thru 0x3c3b95 (0x1cb6 bytes) 000044[0] 1779310933.624868 mapwrite 0x63d9ab thru 0x63e134 (0x78a bytes) 000045[0] 1779310933.676226 write 0x9160cb thru 0x917ca5 (0x1bdb bytes) 000046[1] 1779310933.704001 write 0x4596f8 thru 0x45a71b (0x1024 bytes) 000048[0] 1779310933.734888 write 0x538502 thru 0x539b56 (0x1655 bytes) 000049[1] 1779310933.751752 write 0x3ab227 thru 0x3ab604 (0x3de bytes) truncating to largest ever: 0x9d87ea 000050[0] 1779310933.773666 trunc from 0x917ca6 to 0x9d87e9 (0xc0b44 bytes) 000054[0] 1779310933.836122 trunc from 0x9d87ea to 0x5f2352 (0x3e6497 bytes) 000055[0] 1779310933.861887 write 0x8be399 thru 0x8be6ad (0x315 bytes) 000056[1] 1779310933.863855 mapwrite 0x96fbdf thru 0x9703a9 (0x7cb bytes) 000058[1] 1779310933.985689 trunc from 0x9703aa to 0x5ee1b8 (0x3821f1 bytes) 000059[1] 1779310934.031760 write 0x5da954 thru 0x5db4ef (0xb9c bytes) 000060[0] 1779310934.060145 mapwrite 0x8dced5 thru 0x8de557 (0x1683 bytes) 000063[0] 1779310934.223131 mapwrite 0x1fd267 thru 0x1fd4e6 (0x280 bytes) 000065[0] 1779310934.297643 mapwrite 0x3fa767 thru 0x3fb0aa (0x944 bytes) 000069[0] 1779310934.364655 read 0x055561 thru 0x057019 (0x1ab9 bytes) 000071[1] 1779310934.399237 read 0x8c51b9 thru 0x8c65f2 (0x143a bytes) 000072[0] 1779310934.425026 write 0x73ffb3 thru 0x741193 (0x11e1 bytes) 000073[0] 1779310934.436213 trunc from 0x8de558 to 0x6e455f (0x1f9ff8 bytes) 000075[1] 1779310934.475035 read 0x02bebf thru 0x02cff3 (0x1135 bytes) 000076[1] 1779310934.497095 write 0x24cbfd thru 0x24d81e (0xc22 bytes) 000077[1] 1779310934.522636 mapread 0x671539 thru 0x672f26 (0x19ee bytes) 000079[0] 1779310934.554124 read 0x5b953c thru 0x5baa35 (0x14fa bytes) 000080[1] 1779310934.558447 read 0x602e3f thru 0x604a3a (0x1bfc bytes) 000081[0] 1779310934.563480 mapread 0x51e0d2 thru 0x51ea09 (0x938 bytes) 000082[1] 1779310934.575604 read 0x37d3a3 thru 0x37db19 (0x777 bytes) 000085[1] 1779310934.625410 read 0x4df9d7 thru 0x4e0a58 (0x1082 bytes) 000090[0] 1779310934.656116 write 0x0527d8 thru 0x053db3 (0x15dc bytes) 000091[0] 1779310934.698853 mapwrite 0x8414ea thru 0x841697 (0x1ae bytes) 000092[0] 1779310934.755403 mapwrite 0x6c848e thru 0x6c9497 (0x100a bytes) 000093[0] 1779310934.787425 read 0x26beee thru 0x26c492 (0x5a5 bytes) 000094[1] 1779310934.809678 read 0x4a8738 thru 0x4a9fb2 (0x187b bytes) 000095[0] 1779310934.834139 mapread 0x45d4b0 thru 0x45dd50 (0x8a1 bytes) 000096[0] 1779310934.842414 trunc from 0x841698 to 0x372b4d (0x4ceb4a bytes) 000099[0] 1779310934.884169 mapread 0x250f5d thru 0x252728 (0x17cc bytes) 000101[0] 1779310934.919508 mapwrite 0x4f01ca thru 0x4f167e (0x14b5 bytes) 000103[1] 1779310934.989952 trunc from 0x4f167f to 0x6a348f (0x1b1e11 bytes) 000106[0] 1779310935.066953 read 0x591d97 thru 0x5923f9 (0x663 bytes) 000107[0] 1779310935.091983 mapwrite 0x614714 thru 0x61586b (0x1158 bytes) 000108[1] 1779310935.176653 mapwrite 0x5c9558 thru 0x5ca6a6 (0x114f bytes) 000110[0] 1779310935.257433 trunc from 0x6a3490 to 0x505572 (0x19df1d bytes) 000111[0] 1779310935.319708 write 0x86b20b thru 0x86cd96 (0x1b8c bytes) 000112[1] 1779310935.326588 trunc from 0x86cd97 to 0x8a54c4 (0x3872e bytes) 000114[0] 1779310935.398923 read 0x453633 thru 0x45512d (0x1afb bytes) 000116[0] 1779310935.432228 read 0x54d153 thru 0x54e9eb (0x1899 bytes) 000117[1] 1779310935.442065 write 0x8ad715 thru 0x8ae26f (0xb5b bytes) 000118[0] 1779310935.457961 mapwrite 0x410997 thru 0x410a1a (0x84 bytes) 000121[1] 1779310935.566898 mapwrite 0x2f5a4c thru 0x2f5da5 (0x35a bytes) 000122[0] 1779310935.629425 read 0x00edfa thru 0x00fc61 (0xe68 bytes) 000123[1] 1779310935.642405 read 0x0447c7 thru 0x04654b (0x1d85 bytes) 000124[0] 1779310935.669066 read 0x2e6e9b thru 0x2e8e0c (0x1f72 bytes) 000125[1] 1779310935.673989 trunc from 0x8ae270 to 0x3db65f (0x4d2c10 bytes) 000127[0] 1779310935.721096 write 0x4056ad thru 0x40589e (0x1f2 bytes) 000130[0] 1779310935.782612 read 0x1452ae thru 0x146ce4 (0x1a37 bytes) 000134[1] 1779310935.844112 read 0x24b12e thru 0x24c398 (0x126b bytes) 000135[0] 1779310935.860750 mapwrite 0x2efbe0 thru 0x2f1045 (0x1466 bytes) 000140[1] 1779310935.963055 read 0x2acc44 thru 0x2adb14 (0xed1 bytes) 000143[0] 1779310936.003157 mapread 0x3b2804 thru 0x3b30a3 (0x8a0 bytes) 000144[1] 1779310936.019358 mapread 0x31a53a thru 0x31a7c6 (0x28d bytes) 000145[0] 1779310936.028789 mapread 0x3a548e thru 0x3a6c3c (0x17af bytes) 000146[0] 1779310936.035180 mapwrite 0x4860e8 thru 0x4877a1 (0x16ba bytes) 000147[0] 1779310936.118410 mapwrite 0x75a20c thru 0x75bf35 (0x1d2a bytes) 000149[0] 1779310936.209422 write 0x9ea6fd thru 0x9ec063 (0x1967 bytes) 000151[0] 1779310936.228822 mapwrite 0x547a46 thru 0x548aae (0x1069 bytes) 000153[1] 1779310936.273440 trunc from 0x9ec064 to 0x169c3e (0x882425 bytes) 000155[0] 1779310936.312153 mapwrite 0x65e4a3 thru 0x65e535 (0x93 bytes) 000157[1] 1779310936.428016 trunc from 0x65e536 to 0x2906d8 (0x3cde5d bytes) 000158[1] 1779310936.455498 mapwrite 0x234874 thru 0x2358eb (0x1078 bytes) 000159[0] 1779310936.521419 mapread 0x2476a6 thru 0x247830 (0x18b bytes) 000165[1] 1779310936.575827 mapwrite 0x881744 thru 0x882add (0x139a bytes) 000168[1] 1779310936.667612 mapread 0x1c1dd8 thru 0x1c2cc9 (0xef2 bytes) 000169[0] 1779310936.708184 mapread 0x7b6a57 thru 0x7b86c2 (0x1c6c bytes) 000170[0] 1779310936.748778 mapwrite 0x866faf thru 0x868b89 (0x1bdb bytes) 000171[1] 1779310936.811139 mapwrite 0x9b3b80 thru 0x9b41c9 (0x64a bytes) 000172[1] 1779310936.936243 trunc from 0x9b41ca to 0x78c8f0 (0x2278d9 bytes) 000173[1] 1779310936.961408 trunc from 0x78c8f1 to 0x54e19e (0x23e752 bytes) 000175[1] 1779310936.989626 write 0x0c303c thru 0x0c4d50 (0x1d15 bytes) 000176[1] 1779310937.008196 mapread 0x48d808 thru 0x48f422 (0x1c1b bytes) 000178[1] 1779310937.029916 trunc from 0x54e19f to 0x18a227 (0x3c3f77 bytes) 000180[0] 1779310937.082766 read 0x0ffb34 thru 0x0ffbc5 (0x92 bytes) 000182[0] 1779310937.151749 mapwrite 0x0677cb thru 0x068c14 (0x144a bytes) 000183[1] 1779310937.258698 mapread 0x162f91 thru 0x164c81 (0x1cf1 bytes) 000184[0] 1779310937.302506 mapread 0x112114 thru 0x113fe2 (0x1ecf bytes) 000185[0] 1779310937.339960 read 0x0f9588 thru 0x0fb10e (0x1b87 bytes) 000186[0] 1779310937.369642 mapwrite 0x4e9f98 thru 0x4ea8bd (0x926 bytes) 000188[0] 1779310937.484072 read 0x4abf78 thru 0x4ad0bf (0x1148 bytes) 000190[1] 1779310937.514393 mapread 0x4a2e9b thru 0x4a3f1a (0x1080 bytes) 000191[0] 1779310937.570159 read 0x0f08ac thru 0x0f2329 (0x1a7e bytes) 000192[0] 1779310937.578966 mapwrite 0x030bed thru 0x03157a (0x98e bytes) 000195[1] 1779310937.689199 mapread 0x177b56 thru 0x177c51 (0xfc bytes) 000196[0] 1779310937.726631 trunc from 0x4ea8be to 0x4cfda6 (0x1ab17 bytes) 000197[0] 1779310937.785610 write 0x05f658 thru 0x060808 (0x11b1 bytes) 000199[1] 1779310937.848390 mapwrite 0x414a62 thru 0x415701 (0xca0 bytes) 000200[1] 1779310937.934864 read 0x37c531 thru 0x37d472 (0xf42 bytes) 000202[1] 1779310937.968700 read 0x3a129b thru 0x3a18ad (0x613 bytes) 000203[0] 1779310937.983638 mapwrite 0x9da837 thru 0x9db234 (0x9fe bytes) 000206[1] 1779310938.119609 mapwrite 0x3d3d9b thru 0x3d53bf (0x1625 bytes) 000207[1] 1779310938.206820 trunc from 0x9db235 to 0x6440f2 (0x397142 bytes) 000208[0] 1779310938.249597 mapread 0x1772f4 thru 0x177cac (0x9b9 bytes) 000209[0] 1779310938.294901 mapread 0x2dc885 thru 0x2dd798 (0xf14 bytes) 000210[1] 1779310938.326428 trunc from 0x6440f3 to 0x7eb227 (0x1a7135 bytes) 000219[1] 1779310938.426562 write 0x88abe4 thru 0x88aff3 (0x410 bytes) 000221[1] 1779310938.468852 write 0x42c572 thru 0x42d241 (0xcd0 bytes) 000222[1] 1779310938.493005 trunc from 0x88aff4 to 0x1b6422 (0x6d4bd1 bytes) 000223[1] 1779310938.513210 write 0x065086 thru 0x0663d1 (0x134c bytes) 000224[1] 1779310938.535796 mapwrite 0x901e25 thru 0x903d70 (0x1f4c bytes) 000225[1] 1779310938.632363 read 0x787e10 thru 0x788fa9 (0x119a bytes) 000227[0] 1779310938.657500 trunc from 0x903d71 to 0x33313b (0x5d0c35 bytes) 000232[1] 1779310938.725639 trunc from 0x33313c to 0x24cacb (0xe6670 bytes) 000233[1] 1779310938.772207 write 0x465f00 thru 0x46675c (0x85d bytes) 000234[0] 1779310938.791934 mapwrite 0x213347 thru 0x215146 (0x1e00 bytes) 000236[0] 1779310938.875821 write 0x06f3c4 thru 0x06f9db (0x618 bytes) 000238[0] 1779310938.932441 trunc from 0x46675d to 0x35b8c7 (0x10ae95 bytes) 000239[0] 1779310938.971149 write 0x80a95a thru 0x80bfcb (0x1672 bytes) 000241[1] 1779310939.005371 read 0x4bade6 thru 0x4bc547 (0x1762 bytes) 000247[1] 1779310939.066854 write 0x417121 thru 0x417492 (0x372 bytes) 000248[1] 1779310939.102359 trunc from 0x80bfcc to 0x7d1bcc (0x3a3ff bytes) 000249[0] 1779310939.148918 write 0x3e2898 thru 0x3e32da (0xa43 bytes) 000253[0] 1779310939.243399 write 0x69490e thru 0x695d3f (0x1432 bytes) 000254[1] 1779310939.264773 mapwrite 0x865300 thru 0x865626 (0x327 bytes) 000255[0] 1779310939.360802 mapread 0x54269f thru 0x543f6c (0x18ce bytes) 000256[1] 1779310939.399733 mapread 0x006e8f thru 0x007f98 (0x110a bytes) 000257[0] 1779310939.410692 write 0x521f08 thru 0x522f38 (0x1031 bytes) 000261[0] 1779310939.486481 read 0x7eeb4e thru 0x7eeecf (0x382 bytes) 000264[1] 1779310939.535255 trunc from 0x865627 to 0x610027 (0x2555ff bytes) 000265[1] 1779310939.584257 read 0x1d04dc thru 0x1d11c7 (0xcec bytes) 000268[0] 1779310939.594415 write 0x2f3543 thru 0x2f41ee (0xcac bytes) 000269[1] 1779310939.630872 write 0x1aedb5 thru 0x1afc75 (0xec1 bytes) 000270[1] 1779310939.668430 mapwrite 0x1d4be8 thru 0x1d59af (0xdc8 bytes) 000271[0] 1779310939.733213 read 0x13d6e5 thru 0x13db69 (0x485 bytes) 000277[0] 1779310939.839637 mapread 0x1221c0 thru 0x123b12 (0x1953 bytes) 000278[1] 1779310939.874186 trunc from 0x610028 to 0x94e671 (0x33e64a bytes) 000282[1] 1779310939.965026 mapread 0x3fb76b thru 0x3fbedc (0x772 bytes) 000283[1] 1779310939.972748 mapread 0x7183b6 thru 0x719c66 (0x18b1 bytes) 000286[0] 1779310940.015325 trunc from 0x94e672 to 0x683af3 (0x2cab7e bytes) 000287[1] 1779310940.048860 read 0x47c76c thru 0x47debd (0x1752 bytes) 000289[1] 1779310940.061018 read 0x64ba0c thru 0x64bbe3 (0x1d8 bytes) 000290[1] 1779310940.066047 mapread 0x5c36e3 thru 0x5c3bce (0x4ec bytes) 000291[1] 1779310940.079727 mapread 0x3bab07 thru 0x3bb1af (0x6a9 bytes) 000295[0] 1779310940.115854 mapread 0x28b68b thru 0x28d10b (0x1a81 bytes) 000296[1] 1779310940.128678 read 0x0df723 thru 0x0e0d3b (0x1619 bytes) 000298[1] 1779310940.134721 trunc from 0x683af4 to 0x3f2a6a (0x291089 bytes) 000299[0] 1779310940.165380 write 0x574c75 thru 0x5753e6 (0x772 bytes) 000300[1] 1779310940.185503 mapwrite 0x98d328 thru 0x98d341 (0x1a bytes) 000301[0] 1779310940.275196 mapwrite 0x332a6b thru 0x33441d (0x19b3 bytes) 000305[1] 1779310940.433180 mapread 0x033e10 thru 0x0354e9 (0x16da bytes) 000308[1] 1779310940.503217 trunc from 0x98d342 to 0x130853 (0x85caee bytes) 000309[0] 1779310940.529859 mapwrite 0x10987d thru 0x10a53f (0xcc3 bytes) 000312[0] 1779310940.671222 mapread 0x11b017 thru 0x11cee6 (0x1ed0 bytes) 000313[1] 1779310940.697233 read 0x0a9004 thru 0x0aa4a7 (0x14a4 bytes) 000314[0] 1779310940.738343 write 0x1531c7 thru 0x153b55 (0x98f bytes) 000316[0] 1779310940.781194 mapread 0x13d1c0 thru 0x13e767 (0x15a8 bytes) 000317[1] 1779310940.821924 write 0x95bc12 thru 0x95d6a3 (0x1a92 bytes) 000319[1] 1779310940.841527 mapread 0x8c851e thru 0x8c8dfd (0x8e0 bytes) 000322[1] 1779310940.910411 mapread 0x2c065e thru 0x2c1e0f (0x17b2 bytes) 000324[1] 1779310940.930746 trunc from 0x95d6a4 to 0x4e41ad (0x4794f6 bytes) 000328[1] 1779310940.984117 read 0x1a9e54 thru 0x1ab5ec (0x1799 bytes) 000329[1] 1779310940.984855 read 0x3190e3 thru 0x319a77 (0x995 bytes) 000330[1] 1779310940.999378 read 0x1ddd6c thru 0x1de932 (0xbc7 bytes) 000332[1] 1779310941.037945 mapwrite 0x2e4292 thru 0x2e43a1 (0x110 bytes) 000334[1] 1779310941.112867 trunc from 0x4e41ae to 0x22e89b (0x2b5912 bytes) 000339[0] 1779310941.221095 mapwrite 0x47606e thru 0x4761d4 (0x167 bytes) 000341[0] 1779310941.315083 write 0x0e8a76 thru 0x0e8e9f (0x42a bytes) 000343[0] 1779310941.346869 write 0x6d37fc thru 0x6d3d1c (0x521 bytes) 000344[1] 1779310941.349119 read 0x4a6cae thru 0x4a8648 (0x199b bytes) 000346[1] 1779310941.383961 mapwrite 0x04997a thru 0x04b0e0 (0x1767 bytes) 000348[1] 1779310941.470459 read 0x3c5860 thru 0x3c5b02 (0x2a3 bytes) 000350[1] 1779310941.493462 read 0x22881b thru 0x229f00 (0x16e6 bytes) 000352[1] 1779310941.532918 trunc from 0x6d3d1d to 0x4be3cb (0x215951 bytes) 000353[0] 1779310941.561672 read 0x2d22c1 thru 0x2d3ac3 (0x1803 bytes) truncating to largest ever: 0x9e72ed 000354[0] 1779310941.596778 trunc from 0x4be3cc to 0x9e72ec (0x528f21 bytes) 000355[0] 1779310941.657484 read 0x60132c thru 0x6015bf (0x294 bytes) 000359[0] 1779310941.734385 trunc from 0x9e72ed to 0x484c2d (0x5626bf bytes) 000361[0] 1779310941.757218 trunc from 0x484c2e to 0x178c00 (0x30c02d bytes) 000362[1] 1779310941.811769 read 0x0c9511 thru 0x0caa07 (0x14f7 bytes) 000367[1] 1779310941.883921 trunc from 0x178c01 to 0x142d41 (0x35ebf bytes) 000371[1] 1779310941.969266 trunc from 0x142d42 to 0x29773e (0x1549fd bytes) 000373[1] 1779310942.002867 read 0x15b320 thru 0x15d190 (0x1e71 bytes) 000375[1] 1779310942.039120 write 0x186fff thru 0x18708e (0x90 bytes) 000376[1] 1779310942.054916 mapread 0x1f93ef thru 0x1fa476 (0x1088 bytes) 000379[1] 1779310942.084230 mapread 0x18f98f thru 0x1908c6 (0xf38 bytes) 000382[1] 1779310942.141206 write 0x290b6b thru 0x291570 (0xa06 bytes) 000383[0] 1779310942.142652 write 0x2be2de thru 0x2c0245 (0x1f68 bytes) 000384[0] 1779310942.185940 write 0x0f87eb thru 0x0f97a0 (0xfb6 bytes) 000387[1] 1779310942.256747 write 0x5f5b0c thru 0x5f6591 (0xa86 bytes) 000389[0] 1779310942.287928 write 0x24c5f1 thru 0x24c85b (0x26b bytes) 000390[0] 1779310942.307765 mapwrite 0x130e2c thru 0x132d59 (0x1f2e bytes) 000392[1] 1779310942.421911 trunc from 0x5f6592 to 0x3f4ec2 (0x2016cf bytes) 000394[0] 1779310942.474797 read 0x05b5d0 thru 0x05c2f4 (0xd25 bytes) 000395[1] 1779310942.538707 read 0x0b4347 thru 0x0b527a (0xf34 bytes) 000396[0] 1779310942.598576 mapwrite 0x3ce2ec thru 0x3cf401 (0x1116 bytes) 000400[0] 1779310942.764961 trunc from 0x3f4ec3 to 0x8d9073 (0x4e41b1 bytes) 000401[0] 1779310942.786784 write 0x2b1455 thru 0x2b2b9f (0x174b bytes) 000402[1] 1779310942.799251 trunc from 0x8d9074 to 0x5c09cf (0x3186a4 bytes) 000407[1] 1779310942.906674 mapread 0x190dca thru 0x191af8 (0xd2f bytes) 000410[0] 1779310942.937247 mapread 0x41f07f thru 0x41f0cf (0x51 bytes) 000413[1] 1779310942.997415 read 0x13a6ec thru 0x13aad9 (0x3ee bytes) 000415[0] 1779310943.003205 write 0x589ade thru 0x58a611 (0xb34 bytes) 000416[0] 1779310943.043986 trunc from 0x5c09d0 to 0x666593 (0xa5bc4 bytes) 000418[0] 1779310943.095045 trunc from 0x666594 to 0x1cf366 (0x49722d bytes) 000424[0] 1779310943.190745 read 0x12a743 thru 0x12bdeb (0x16a9 bytes) 000425[0] 1779310943.214272 write 0x56195e thru 0x562d53 (0x13f6 bytes) 000426[0] 1779310943.231326 mapread 0x168f4c thru 0x16a245 (0x12fa bytes) 000427[1] 1779310943.232224 trunc from 0x562d54 to 0x3be4fe (0x1a4855 bytes) 000429[0] 1779310943.300797 mapread 0x2b3604 thru 0x2b4799 (0x1196 bytes) 000439[0] 1779310943.496530 mapread 0x328b74 thru 0x32996f (0xdfc bytes) 000443[1] 1779310943.551182 write 0x2e4800 thru 0x2e55b5 (0xdb6 bytes) 000444[1] 1779310943.576051 mapread 0x13397e thru 0x134bf0 (0x1273 bytes) 000445[0] 1779310943.605123 write 0x0816a3 thru 0x082c59 (0x15b7 bytes) 000448[1] 1779310943.686252 mapwrite 0x3a6b10 thru 0x3a7543 (0xa34 bytes) 000449[0] 1779310943.759264 mapread 0x39c615 thru 0x39d5cc (0xfb8 bytes) 000450[1] 1779310943.784807 trunc from 0x3be4ff to 0x66c20c (0x2add0e bytes) 000453[0] 1779310943.834993 mapread 0x18777c thru 0x18961e (0x1ea3 bytes) 000455[0] 1779310943.873456 write 0x3146b2 thru 0x315e08 (0x1757 bytes) 000458[0] 1779310943.961989 write 0x1d2e5a thru 0x1d48c2 (0x1a69 bytes) 000459[1] 1779310943.995152 write 0x5a63bb thru 0x5a73e2 (0x1028 bytes) 000460[1] 1779310944.026884 mapread 0x3310e7 thru 0x33182d (0x747 bytes) 000461[0] 1779310944.066068 trunc from 0x66c20d to 0x31344c (0x358dc0 bytes) 000463[0] 1779310944.138964 trunc from 0x31344d to 0x2483f2 (0xcb05a bytes) 000464[1] 1779310944.178582 mapread 0x00d1af thru 0x00d9a3 (0x7f5 bytes) 000466[0] 1779310944.226471 mapread 0x11b940 thru 0x11c7f6 (0xeb7 bytes) 000469[1] 1779310944.251907 write 0x7b06d9 thru 0x7b0bb5 (0x4dd bytes) 000471[0] 1779310944.292987 trunc from 0x7b0bb6 to 0x991cd7 (0x1e1122 bytes) 000472[1] 1779310944.358024 write 0x803652 thru 0x803db9 (0x768 bytes) 000473[0] 1779310944.410616 trunc from 0x991cd8 to 0x5c8600 (0x3c96d7 bytes) 000475[0] 1779310944.463366 write 0x68f6c6 thru 0x69050f (0xe4a bytes) 000476[1] 1779310944.465178 read 0x3f1088 thru 0x3f15c3 (0x53c bytes) 000480[0] 1779310944.548454 mapwrite 0x94f4b7 thru 0x951016 (0x1b60 bytes) 000482[1] 1779310944.721061 mapread 0x2c9d42 thru 0x2cab14 (0xdd3 bytes) 000483[0] 1779310944.762217 write 0x1742af thru 0x1742e4 (0x36 bytes) 000486[1] 1779310944.796515 trunc from 0x951017 to 0x83f8e6 (0x111730 bytes) 000487[0] 1779310944.866145 write 0x715534 thru 0x7159bb (0x488 bytes) 000489[0] 1779310944.907276 trunc from 0x83f8e7 to 0x6074fd (0x2383e9 bytes) 000490[1] 1779310944.935895 trunc from 0x6074fe to 0x5d4dfc (0x32701 bytes) 000491[1] 1779310944.974148 write 0x063215 thru 0x064158 (0xf44 bytes) 000493[0] 1779310945.040079 mapwrite 0x39b118 thru 0x39cacb (0x19b4 bytes) 000495[1] 1779310945.167650 mapwrite 0x9abd17 thru 0x9ac745 (0xa2f bytes) 000496[0] 1779310945.272710 mapread 0x6694d3 thru 0x669672 (0x1a0 bytes) 000498[0] 1779310945.324335 mapread 0x2e899a thru 0x2e8a1c (0x83 bytes) 000499[0] 1779310945.354436 write 0x42443d thru 0x42584d (0x1411 bytes) 000500[1] 1779310945.395008 read 0x480cc8 thru 0x481083 (0x3bc bytes) All operations completed A-OK! PASS 16b (18s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 17:02:28 (1779310948) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 17:02:30 (1779310950) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.289919 s, 28.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.242598 s, 34.6 MB/s ldlm.namespaces.MGC192.168.203.135@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff936b09417800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff936b107b5800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff936b09417800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff936b09417800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.50993 s, 16.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.92496 s, 54.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.32856 s, 78.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.44677 s, 30.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.58835 s, 22.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.70322 s, 61.6 MB/s PASS 16d (27s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 17:02:57 (1779310977) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.167002 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852462 s, 12.3 MB/s PASS 16e (4s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 17:03:01 (1779310981) TIMEOUT PASS 16f (25s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 17:03:26 (1779311006) TIMEOUT PASS 16g (24s) == sanityn test 16h: mmap read after truncate file ======= 17:03:50 (1779311030) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (5s) == sanityn test 16i: read after truncate file ============ 17:03:55 (1779311035) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (3s) == sanityn test 16j: race dio with buffered i/o ========== 17:03:58 (1779311038) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.10323 s, 23.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.675304 s, 38.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21951 s, 21.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.728869 s, 36.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.10147 s, 23.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.933976 s, 28.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.21676 s, 21.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.07874 s, 24.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08768 s, 24.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.38601 s, 18.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.49051 s, 17.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05016 s, 25.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08652 s, 24.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.30326 s, 20.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.41687 s, 18.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.35737 s, 19.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.57207 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.7994 s, 14.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.87443 s, 14.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (19s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 17:04:17 (1779311057) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! + fsx 1 done: rc=0 Wed May 20 17:04:35 EDT 2026 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Wed May 20 17:04:36 EDT 2026 + fsx 3 done: rc=0 Wed May 20 17:04:36 EDT 2026 + fsx 4 done: rc=0 Wed May 20 17:04:36 EDT 2026 + fsx 5 done: rc=0 Wed May 20 17:04:36 EDT 2026 PASS 16k (23s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 17:04:40 (1779311080) fail_loc=0x8000030a PASS 17 (6s) == sanityn test 18: mmap sanity check =========================================================================================== 17:04:46 (1779311086) mmap test1: basic mmap operation (PASS, 0.081915s) mmap test2: MAP_PRIVATE not write back (PASS, 0.037559s) mmap test3: concurrent mmap ops on two nodes (PASS, 10.743s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.3394s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.12617s) mmap test6: check mmap write/read content on two nodes (PASS, 0.13167s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.71944s) mmap test11: multi-thread shared mmap (PASS, 0.73707s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0632s) PASS 18 (23s) == sanityn test 19: test concurrent uncached read races ========================================================================= 17:05:09 (1779311109) oleg335-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (3s) == sanityn test 20: test extra readahead page left in cache ============================================================== 17:05:12 (1779311112) PASS 20 (4s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 17:05:16 (1779311116) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (4s) == sanityn test 23: others should see updated atime while another read============================================================== 17:05:20 (1779311120) now is 1779311121 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 new atime is 1779311182 PASS 23 (65s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 17:06:25 (1779311185) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 5.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 37.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 403 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 400 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118099 403 117696 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117232 400 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234914 386 234528 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 5120 3764224 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 PASS 24a (3s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 17:06:28 (1779311188) PASS 24b (4s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 17:06:32 (1779311192) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (3s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 17:06:35 (1779311195) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (2s) == sanityn test 26a: allow mtime to get older ============ 17:06:37 (1779311197) utime: good mknod times 1779311197 <= 1779311198 <= 1779311198 for /mnt/lustre/f26a utime: good mknod times 1779311197 <= 1779311198 <= 1779311198 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (4s) == sanityn test 26b: sync mtime between ost and mds ====== 17:06:41 (1779311201) PASS 26b (5s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 17:06:46 (1779311206) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-20 17:06:47.000000000 -0400 Birth: 2026-05-20 17:06:47.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-05-20 17:06:47.000000000 -0400 Birth: 2026-05-20 17:06:47.000000000 -0400 PASS 26c (5s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 17:06:51 (1779311211) 0 PASS 30 (6s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 17:06:57 (1779311217) fail_loc=0x314 PASS 31a (4s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 17:07:01 (1779311221) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.135@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (15s) == sanityn test 31r: open-rename(replace) race =========== 17:07:16 (1779311236) -rw-r--r-- 1 root root 0 May 20 17:07 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 17:07:22 (1779311242) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43012 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 17:07:27 (1779311247) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 17:07:32 (1779311252) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 17:07:33 (1779311253) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 17:07:35 (1779311255) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 17:07:37 (1779311257) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 17:07:38 (1779311258) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid in FULL state after 0 sec oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid in IDLE state after 0 sec oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid in FULL state after 0 sec oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid in IDLE state after 0 sec oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936b09417800.ost_server_uuid in FULL state after 0 sec oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff936b09417800.ost_server_uuid in IDLE state after 0 sec PASS 34 (101s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 17:09:19 (1779311359) fail_loc=0x80000317 Race attempt 0 Wait for 49062 49168 for 60 sec... - open/close 1168 (time 1779311372.95 total 10.00 last 116.79) - open/close 2491 (time 1779311382.95 total 20.00 last 132.28) - open/close 3925 (time 1779311392.95 total 30.00 last 143.40) total: 4000 open/close in 30.57 seconds: 130.83 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.135@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (67s) == sanityn test 36: handle ESTALE/open-unlink correctly == 17:10:26 (1779311426) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg335-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.545643 s, 96.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.642314 s, 81.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.52454 s, 100 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.637303 s, 82.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.564043 s, 93.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.573453 s, 91.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7493632) after_dd(7441408) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.572295 s, 91.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7441408) after(7492608) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.595713 s, 88.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7492608) after_dd(7443456) after(7494656) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.503606 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7494656) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.499761 s, 105 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7496704) after_dd(7445504) after(7496704) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.559131 s, 93.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7496704) after_dd(7443456) after(7496704) PASS 36 (260s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 17:14:47 (1779311687) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 - create 606 (time 1779311708.83 total 10.02 last 60.46) - create 1264 (time 1779311718.84 total 20.04 last 65.71) - create 2108 (time 1779311728.84 total 30.04 last 84.37) - create 2941 (time 1779311738.85 total 40.05 last 83.24) - create 4197 (time 1779311748.85 total 50.05 last 125.55) - create 5361 (time 1779311758.86 total 60.06 last 116.36) - create 6502 (time 1779311768.87 total 70.06 last 113.99) - create 7597 (time 1779311778.88 total 80.07 last 109.40) - create 8741 (time 1779311788.89 total 90.08 last 114.30) - create 9936 (time 1779311798.89 total 100.09 last 119.42) total: 10000 create in 100.56 seconds: 99.45 ops/second PASS 37 (118s) == sanityn test 39a: file mtime does not change after rename ========================================================== 17:16:46 (1779311806) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 17:16:52 (1779311812) repeat after cancel_lru_locks PASS 39b (7s) == sanityn test 39c: check truncate mtime update ================================================================================ 17:16:59 (1779311819) repeat after cancel_lru_locks PASS 39c (8s) == sanityn test 39d: sync write should update mtime ====== 17:17:07 (1779311827) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 17:17:16 (1779311836) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 17:17:34 (1779311854) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (19s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 17:17:53 (1779311873) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 17:18:11 (1779311891) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 17:18:29 (1779311909) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 17:18:43 (1779311923) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 17:18:57 (1779311937) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (15s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 17:19:12 (1779311952) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 17:19:27 (1779311967) Conflict PASS 41e (15s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 17:19:42 (1779311982) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 17:19:57 (1779311997) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 17:20:12 (1779312012) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (15s) == sanityn test 41i: reint_open: create vs create ======== 17:20:28 (1779312028) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1015s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 17:37:23 (1779313043) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (11s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 17:37:34 (1779313054) open: File exists Conflict PASS 42b (12s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 17:37:46 (1779313066) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (11s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 17:37:57 (1779313077) Conflict PASS 42d (11s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 17:38:08 (1779313088) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (13s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 17:38:21 (1779313101) Conflict PASS 42f (11s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 17:38:32 (1779313112) Conflict PASS 42g (12s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 17:38:45 (1779313125) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 17:38:56 (1779313136) PASS 43a (60s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 17:39:56 (1779313196) Conflict PASS 43b (11s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 17:40:07 (1779313207) Conflict PASS 43c (10s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 17:40:17 (1779313217) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (11s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 17:40:29 (1779313229) Conflict PASS 43e (11s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 17:40:39 (1779313239) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (9s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 17:40:48 (1779313248) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (9s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 17:40:57 (1779313257) Conflict PASS 43h (10s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 17:41:07 (1779313267) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (2s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 17:41:09 (1779313269) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (84s) == sanityn test 43k: unlink vs create ==================== 17:42:33 (1779313353) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1002s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 17:59:15 (1779314355) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed May 20 17:59:21 EDT 2026 PASS 44a (10s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 17:59:25 (1779314365) open: File exists Conflict PASS 44b (10s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 17:59:35 (1779314375) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (9s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 17:59:44 (1779314384) Conflict PASS 44d (11s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 17:59:55 (1779314395) Conflict PASS 44e (12s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 18:00:08 (1779314408) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 18:00:23 (1779314423) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 18:00:37 (1779314437) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 18:00:50 (1779314450) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 18:00:53 (1779314453) PASS 45a (136s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 18:03:09 (1779314589) Conflict PASS 45b (13s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 18:03:22 (1779314602) Conflict PASS 45c (17s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 18:03:39 (1779314619) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 18:03:52 (1779314632) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 18:04:09 (1779314649) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (16s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 18:04:26 (1779314666) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 18:04:41 (1779314681) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 18:04:55 (1779314695) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 18:04:59 (1779314699) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1062s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 18:22:42 (1779315762) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (10s) == sanityn test 46b: pdirops: link vs create ======================================================================== 18:22:53 (1779315773) open: File exists Conflict PASS 46b (11s) == sanityn test 46c: pdirops: link vs link ======================================================================== 18:23:04 (1779315784) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 18:23:16 (1779315796) Conflict PASS 46d (11s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 18:23:27 (1779315807) Conflict PASS 46e (12s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 18:23:39 (1779315819) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 18:23:51 (1779315831) Conflict PASS 46g (10s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 18:24:02 (1779315842) Conflict PASS 46h (11s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 18:24:13 (1779315853) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 18:24:16 (1779315856) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (2s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 18:24:19 (1779315859) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 18:24:21 (1779315861) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 18:24:24 (1779315864) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 18:24:27 (1779315867) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (2s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 18:24:30 (1779315870) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (2s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 18:24:32 (1779315872) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 18:24:34 (1779315874) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.123495 s, 82.9 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (10s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 18:24:44 (1779315884) 0+1 records in 0+1 records out 158 bytes copied, 0.0227414 s, 6.9 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217574) - No such process PASS 51a (7s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 18:24:51 (1779315891) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00925104 s, 111 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555385 s, 18.9 MB/s 3145728 PASS 51b (17s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 18:25:08 (1779315908) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.018649 s, 54.9 kB/s PASS 51c (10s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 18:25:18 (1779315918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304123 s, 34.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (5s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 18:25:23 (1779315923) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2453 0x995 0x240000400 read lease(1) has applied. PASS 51e (6s) == sanityn test 54: rename locking ======================= 18:25:29 (1779315929) llite.lustre-ffff936b09417800.xattr_cache=1 llite.lustre-ffff936b107b5800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:30.000000000 -0400 Modify: 2026-05-20 18:25:30.000000000 -0400 Change: 2026-05-20 18:25:30.000000000 -0400 Birth: 2026-05-20 18:25:30.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:36.000000000 -0400 Modify: 2026-05-20 18:25:37.000000000 -0400 Change: 2026-05-20 18:25:37.000000000 -0400 Birth: 2026-05-20 18:25:36.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:42.000000000 -0400 Modify: 2026-05-20 18:25:42.000000000 -0400 Change: 2026-05-20 18:25:42.000000000 -0400 Birth: 2026-05-20 18:25:42.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:42.000000000 -0400 Modify: 2026-05-20 18:25:43.000000000 -0400 Change: 2026-05-20 18:25:43.000000000 -0400 Birth: 2026-05-20 18:25:42.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:48.000000000 -0400 Modify: 2026-05-20 18:25:49.000000000 -0400 Change: 2026-05-20 18:25:49.000000000 -0400 Birth: 2026-05-20 18:25:48.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:25:48.000000000 -0400 Modify: 2026-05-20 18:25:49.000000000 -0400 Change: 2026-05-20 18:25:49.000000000 -0400 Birth: 2026-05-20 18:25:48.000000000 -0400 PASS 54 (28s) == sanityn test 55a: rename vs unlink target dir ========= 18:25:57 (1779315957) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (11s) == sanityn test 55b: rename vs unlink source dir ========= 18:26:09 (1779315969) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (10s) == sanityn test 55c: rename vs unlink orphan target dir == 18:26:19 (1779315979) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (16s) == sanityn test 55d: rename file vs link ================= 18:26:35 (1779315995) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (12s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 18:26:47 (1779316007) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (2s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 18:26:49 (1779316009) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (15s) == sanityn test 55g: rename: race with trylock =========== 18:27:04 (1779316024) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (18s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 18:27:22 (1779316042) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1779316043 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.753569 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (85s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 18:28:47 (1779316127) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1779316129 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 2.44148 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (56s) == sanityn test 60: Verify data_version behaviour ======== 18:29:43 (1779316183) PASS 60 (4s) == sanityn test 70a: cd directory && rm directory ======== 18:29:48 (1779316188) PASS 70a (4s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 18:29:52 (1779316192) PASS 70b (5s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 18:29:57 (1779316197) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00609185 s, 10.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (2s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 18:29:59 (1779316199) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0161029 s, 4.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (2s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 18:30:01 (1779316201) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 18:30:03 (1779316203) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0130336 s, 5.0 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 18:30:04 (1779316204) llite.lustre-ffff936b09417800.xattr_cache=1 llite.lustre-ffff936b107b5800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (4s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 18:30:08 (1779316208) llite.lustre-ffff936b09417800.xattr_cache=1 llite.lustre-ffff936b107b5800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (4s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 18:30:12 (1779316212) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0108407 s, 94.5 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00638009 s, 160 kB/s 230360: get lock2 230361: get lock1 230361: done 230361: get lock2 230360: done 230360: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 230361: cannot get lock2: Resource deadlock avoided 230361: exit rc=0 230360: done 230360: exit rc=0 PASS 74 (8s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 18:30:20 (1779316220) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0444387 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238844 s, 43.9 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230941 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (9s) == sanityn test 76: Verify MDT open_files listing ======== 18:30:29 (1779316229) opening files: ........ closing files: ........ PASS 76 (40s) == sanityn test 77a: check FIFO NRS policy =============== 18:31:09 (1779316269) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.215431 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493762 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089234 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962025 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799576 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660114 s, 15.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805768 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101072 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112435 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122276 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128608 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122867 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104928 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960092 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131782 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102422 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147471 s, 7.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315286 s, 33.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447689 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349762 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385179 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490884 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05078 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496305 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260088 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287161 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491773 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432363 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371549 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538802 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365327 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367319 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480386 s, 21.8 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (5s) == sanityn test 77b: check CRR-N NRS policy ============== 18:31:14 (1779316274) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.217348 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419854 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433731 s, 24.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925665 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992822 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959775 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104651 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967349 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117066 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118427 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131849 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116869 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128703 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130041 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112482 s, 9.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120066 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128522 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021487 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225617 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261583 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202199 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248097 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025527 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041076 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299705 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404566 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356388 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041097 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411488 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500512 s, 21.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504855 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359198 s, 29.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.201961 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437995 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563302 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449669 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699415 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742787 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822601 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842539 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992828 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862294 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935527 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101893 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721549 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839338 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103124 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127323 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112085 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320538 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170607 s, 61.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313762 s, 33.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256868 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156918 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168287 s, 62.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159314 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182796 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261119 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025119 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290764 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268278 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313995 s, 33.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387571 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308561 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322897 s, 32.5 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (6s) == sanityn test 77c: check ORR NRS policy ================ 18:31:20 (1779316280) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.244927 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481255 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738405 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865188 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789805 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882888 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891498 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108164 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996727 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481504 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108952 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125094 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979382 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059663 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107564 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102932 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122127 s, 8.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279248 s, 37.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284078 s, 36.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313055 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251132 s, 41.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451761 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272868 s, 38.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386168 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219111 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031592 s, 33.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041912 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312888 s, 33.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372024 s, 28.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431484 s, 24.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510567 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296754 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034719 s, 30.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246165 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038244 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436089 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104944 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849522 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105793 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121667 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103814 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146088 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119485 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=6 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149587 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160072 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159574 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151342 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147709 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155185 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176794 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869099 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911549 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983592 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769268 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915611 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089154 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888291 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831017 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905023 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913978 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810771 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930917 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106798 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995899 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960964 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981789 s, 10.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.241993 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586207 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973313 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102211 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125459 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132984 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118884 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110201 s, 9.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977235 s, 10.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107956 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10955 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133702 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136318 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123981 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109136 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124906 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134985 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142636 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236287 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00407352 s, 257 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381224 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476834 s, 22.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531455 s, 19.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148211 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482571 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134136 s, 78.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358692 s, 29.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256937 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00551833 s, 190 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00605504 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00359476 s, 292 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581063 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00370486 s, 283 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (9s) == sanityn test 77d: check TRR nrs policy ================ 18:31:29 (1779316289) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.256715 s, 65.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412785 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058619 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867138 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108882 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116004 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128247 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13808 s, 7.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139709 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156956 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146931 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15634 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139493 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133861 s, 7.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135777 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16616 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166185 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018988 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312632 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325813 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494029 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298322 s, 35.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315263 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396993 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333555 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384663 s, 27.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440351 s, 23.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055164 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451252 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397754 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554412 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399174 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521859 s, 20.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.274491 s, 61.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635632 s, 16.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768011 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449708 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638065 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691856 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885677 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907933 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626618 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133815 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106437 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142899 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154065 s, 6.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127474 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144585 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145962 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182509 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114289 s, 91.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329622 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251775 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00443564 s, 236 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00567971 s, 185 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113839 s, 92.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011665 s, 89.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00523577 s, 200 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00571931 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050872 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0110762 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335911 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04502 s, 23.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00403672 s, 260 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483572 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425857 s, 246 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.264042 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045764 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513324 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130601 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156823 s, 6.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148593 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130617 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138201 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132034 s, 7.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141325 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135656 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120757 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133817 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135549 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154553 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140221 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157362 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071982 s, 14.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643094 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143963 s, 72.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167026 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792861 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223727 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264329 s, 39.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365977 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265794 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0854981 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282999 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772458 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800751 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328515 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927427 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344066 s, 30.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (9s) == sanityn test 77e: check TBF NID nrs policy ============ 18:31:38 (1779316298) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.35@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.261606 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475752 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411796 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705867 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120253 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172804 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186239 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159179 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19539 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208145 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213856 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219947 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240971 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200052 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21087 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248356 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252337 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158266 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158986 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152145 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155243 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16015 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16717 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160803 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163081 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158165 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172496 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167867 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177624 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194731 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175197 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179638 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177719 s, 5.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.252795 s, 66.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0376974 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390152 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151261 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152137 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126088 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139577 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162805 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141088 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151624 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145849 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184056 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175984 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177785 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182796 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19575 s, 5.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168093 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172089 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169865 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163372 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172916 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169733 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170594 s, 6.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170943 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17514 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178952 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189528 s, 5.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171548 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178078 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175684 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19343 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185325 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182849 s, 5.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.26495 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550743 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582304 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546727 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477032 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828263 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078219 s, 13.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771726 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060653 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585074 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106133 s, 9.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124389 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139813 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161129 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138518 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159908 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183745 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233303 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671379 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132567 s, 79.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105722 s, 99.2 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033275 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238882 s, 43.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395687 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362395 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369939 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403696 s, 26.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513217 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433974 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622832 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044094 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464469 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483669 s, 21.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.234173 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540218 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481304 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142115 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11578 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147331 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127229 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159095 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173983 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141321 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19043 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18946 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179313 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179741 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176886 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179371 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188962 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231201 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247118 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270845 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294891 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259552 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412043 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296497 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0522605 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425192 s, 24.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354352 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375724 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424838 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369088 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0391511 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431501 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445105 s, 23.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (13s) == sanityn test 77f: check TBF JobID nrs policy ========== 18:31:51 (1779316311) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.196201 s, 85.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360259 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036964 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403939 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102415 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112194 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110499 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112776 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122324 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141362 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130189 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151622 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134652 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159256 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13738 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149172 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155807 s, 6.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422455 s, 24.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330224 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375626 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362889 s, 28.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437776 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333881 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478266 s, 21.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433122 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488597 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419166 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480711 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573856 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518179 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476009 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535473 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497091 s, 21.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.257558 s, 65.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324881 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359775 s, 29.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121042 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0856029 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116486 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878866 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822049 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093154 s, 11.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152579 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153129 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116095 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127498 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114786 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136576 s, 7.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114578 s, 9.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167541 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198086 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262249 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290333 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025685 s, 40.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024399 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349378 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475481 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350652 s, 29.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405249 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384379 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039851 s, 26.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04384 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331638 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409408 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363883 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518297 s, 20.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.236852 s, 70.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548939 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648252 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654795 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861674 s, 12.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150394 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158523 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133077 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172236 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132568 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187499 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145152 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165243 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166348 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157377 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149544 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173525 s, 6.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100227 s, 105 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00679891 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265866 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00360821 s, 291 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00662847 s, 158 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0074694 s, 140 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00691027 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150701 s, 69.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.013253 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439406 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602451 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491791 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382041 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526534 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784816 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690027 s, 152 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.246741 s, 68.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471763 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0444429 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570965 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619427 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108669 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885824 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140161 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128583 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161756 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157499 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141892 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14979 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158922 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186124 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158863 s, 6.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff936b107b5800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff936b107b5800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213101 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242564 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380677 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455005 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409106 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00585043 s, 179 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00745166 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00748708 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321328 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327905 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398772 s, 26.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349674 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210564 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00366759 s, 286 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122616 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011758 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653038 s, 16.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (13s) == sanityn test 77g: Change TBF type directly ============ 18:32:04 (1779316324) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (6s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 18:32:10 (1779316330) oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg335-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg335-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg335-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg335-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 PASS 77h (7s) == sanityn test 77i: Change rank of TBF rule ============= 18:32:17 (1779316337) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (13s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 18:32:30 (1779316350) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33744 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531967 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060122 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400003 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082927 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652234 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669084 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832754 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801717 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523713 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116563 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136222 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148147 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132047 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140432 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191708 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180319 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033718 s, 31.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136308 s, 76.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476651 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511708 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369785 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041725 s, 25.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450136 s, 23.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432005 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349614 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431155 s, 24.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441316 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310189 s, 33.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493801 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437492 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382758 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473984 s, 22.1 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.69008 s, 18.4 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5183 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (52s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 18:33:22 (1779316402) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.25912 s, 13.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406194 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060055 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570307 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961112 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10407 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971317 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703804 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776792 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151838 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152182 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182276 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185524 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208523 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236675 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223092 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22124 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238135 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240613 s, 43.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405598 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372492 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200266 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216144 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236248 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447549 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034763 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292875 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537699 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479717 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448984 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370754 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587142 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495446 s, 21.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.803 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9283 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2814 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103338 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855994 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846497 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13233 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140761 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151993 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158062 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159582 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112726 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149858 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159505 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147547 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172178 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194388 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18264 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192472 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373827 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035744 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475183 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269696 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428909 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488495 s, 21.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402786 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564219 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412594 s, 25.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621233 s, 16.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417524 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326725 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615283 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362293 s, 28.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0373715 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417226 s, 25.1 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7991 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8933 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (137s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 18:35:39 (1779316539) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.24795 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.074236 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695868 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537943 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997636 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100192 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107613 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748059 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741959 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139336 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13937 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160546 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134913 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132754 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141894 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18908 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197724 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295358 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431304 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396487 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386208 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392446 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447595 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379772 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377001 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491088 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048878 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414314 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381978 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514215 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420813 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0470732 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640254 s, 16.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7872 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9015 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.2556 s, 13.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709488 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070622 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941775 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100423 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908785 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114982 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836808 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120994 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132578 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142278 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16899 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181582 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218658 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202004 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194434 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232622 s, 4.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432731 s, 24.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395901 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404791 s, 25.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509048 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509893 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387219 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04352 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449539 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387185 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591538 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428566 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538096 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518501 s, 20.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492496 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479996 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551954 s, 19.0 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8285 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9149 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (138s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 18:37:57 (1779316677) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.319442 s, 52.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323943 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377611 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120931 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142054 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179546 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168686 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174764 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186511 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167272 s, 6.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174734 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183444 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183508 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178577 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178419 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169848 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179483 s, 5.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312667 s, 33.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284853 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338386 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355489 s, 29.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508563 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354576 s, 29.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514431 s, 20.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054758 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406967 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415848 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448214 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432625 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448252 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453302 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474527 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503583 s, 20.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03846 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77937 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.361828 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445707 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425281 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112107 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979096 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126024 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114004 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125518 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147729 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143021 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161704 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147259 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139869 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184659 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14333 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204014 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2037 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202136 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345747 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015708 s, 66.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244149 s, 42.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019091 s, 54.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213519 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224151 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387608 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245345 s, 42.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408681 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255474 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284087 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528665 s, 19.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320176 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326919 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390441 s, 26.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.16833 s, 20.3 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80163 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.349997 s, 47.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322954 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336289 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523347 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111387 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123001 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132638 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117901 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135426 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145832 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118488 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13265 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141227 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142912 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131729 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13766 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149134 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263856 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275545 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277336 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177623 s, 59.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436974 s, 24.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302409 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369313 s, 28.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396626 s, 26.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394736 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300766 s, 34.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273544 s, 38.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329746 s, 31.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424132 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393613 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324595 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387003 s, 27.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17085 s, 20.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94358 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.559822 s, 30.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546625 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572937 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657675 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112584 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993525 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123348 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981471 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926988 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127681 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134854 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12642 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149332 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761045 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152665 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428459 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42183 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399437 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289647 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295581 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0429459 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314158 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314065 s, 33.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457603 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356766 s, 29.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581765 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372656 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385592 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385083 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423661 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411582 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421129 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463445 s, 22.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98987 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82737 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29584 s, 19.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.80711 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.880041 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423501 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607933 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107928 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122737 s, 8.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106253 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128926 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101548 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117237 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139394 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189576 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206879 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215345 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193388 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23428 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210852 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241335 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163089 s, 64.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263296 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544994 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029312 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406185 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310918 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517031 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234342 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566244 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347731 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371055 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411606 s, 25.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397921 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431714 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394621 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469936 s, 22.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.617 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4909 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.297543 s, 56.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375566 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346177 s, 30.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378358 s, 27.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460035 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519502 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497061 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662406 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667857 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479433 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449543 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115142 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823755 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122908 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125187 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113899 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144166 s, 7.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153778 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019314 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168077 s, 62.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190762 s, 55.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328498 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367394 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252014 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265361 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365291 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271363 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271442 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279649 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336362 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034304 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304568 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383371 s, 27.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99343 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.79076 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.475239 s, 35.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342904 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458621 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134408 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119288 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135394 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107866 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118882 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126758 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137962 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13567 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140228 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127751 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212696 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193079 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245372 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26079 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295997 s, 35.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127021 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172554 s, 60.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281123 s, 37.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352366 s, 29.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241866 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046434 s, 22.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330327 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336517 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277098 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510561 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258268 s, 40.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343508 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323917 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031076 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039907 s, 26.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93909 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78785 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84359 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78299 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (372s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 18:44:09 (1779317049) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.361761 s, 46.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293167 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390273 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887912 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119269 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136524 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13335 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131401 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123702 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138295 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123442 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132587 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143125 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130494 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126564 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170145 s, 6.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244384 s, 42.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150164 s, 69.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212039 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294171 s, 35.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226523 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197156 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360993 s, 29.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195706 s, 53.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227057 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225033 s, 46.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258813 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260685 s, 40.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275561 s, 38.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270882 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285961 s, 36.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292981 s, 35.8 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02693 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.78854 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (38s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 18:44:47 (1779317087) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.35@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.35@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.347101 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311108 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038022 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302414 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564992 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637885 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678481 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436737 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10472 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902384 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102269 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109316 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129897 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19393 s, 5.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197756 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219636 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208029 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319013 s, 32.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343862 s, 30.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254592 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448614 s, 23.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385306 s, 27.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372962 s, 28.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420386 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495629 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325023 s, 32.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371307 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375238 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448024 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439781 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437272 s, 24.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432387 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046034 s, 22.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.03993 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.81241 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (41s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 18:45:29 (1779317129) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.35@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.275816 s, 60.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384631 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660608 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857219 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123437 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125415 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119828 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989997 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102336 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119417 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10736 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128375 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129011 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133277 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116105 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132231 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135272 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285223 s, 36.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222184 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322979 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365832 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374538 s, 28.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246768 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335009 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486766 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347748 s, 30.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457912 s, 22.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314554 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035367 s, 29.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420157 s, 25.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0431291 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378512 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445427 s, 23.5 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99902 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95506 s, 21.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (34s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 18:46:02 (1779317162) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.453257 s, 37.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296142 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0488838 s, 21.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101458 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092396 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111715 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924612 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108973 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994153 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101571 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112891 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963032 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104756 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240837 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24707 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291503 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320533 s, 3.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168185 s, 62.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021969 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137105 s, 76.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269104 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291328 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318795 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298661 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212005 s, 49.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195022 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309563 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00468802 s, 224 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306949 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330683 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00570364 s, 184 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122141 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00233124 s, 450 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84432 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77716 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05192 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7971 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (82s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 18:47:25 (1779317245) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.910551 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377473 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255036 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582454 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818608 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897117 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777226 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747453 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751274 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145103 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187449 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173521 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179133 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183589 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208211 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18088 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198022 s, 5.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261097 s, 40.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286531 s, 36.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310376 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341477 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032427 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416115 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342324 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356044 s, 29.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320811 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329113 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367979 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368463 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345382 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384436 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380454 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367148 s, 28.6 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.57 s, 5.4 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9506 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 18:48:30 (1779317310) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.304573 s, 55.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438659 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445631 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539234 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098496 s, 10.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124122 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108054 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130905 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896626 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114 s, 9.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102435 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136124 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13718 s, 7.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10893 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134787 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119997 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131942 s, 7.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271202 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304924 s, 34.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202831 s, 51.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037631 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137913 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320835 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371333 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234415 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397068 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300635 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330973 s, 31.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381691 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410338 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308275 s, 34.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390663 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114754 s, 91.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.01232 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77879 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg335-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.477678 s, 35.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348636 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508798 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659521 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876826 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108248 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979896 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942008 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130957 s, 8.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123451 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103046 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119749 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991514 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29657 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271544 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361265 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403415 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344149 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214984 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024104 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252911 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403115 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360453 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302378 s, 34.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329238 s, 31.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332257 s, 31.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035958 s, 29.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0530257 s, 19.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0382539 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394432 s, 26.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348646 s, 30.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405436 s, 25.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407206 s, 25.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88681 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.7715 s, 10.7 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.86472 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74598 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (120s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 18:50:30 (1779317430) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:) Stopping client oleg335-client.virtnet /mnt/lustre opts: Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg335-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg335-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg335-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg335-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.35,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg335-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.35,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/64045/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64-retry1 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.35,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg335-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre Starting client oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Started clients oleg335-client.virtnet: 192.168.203.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff936b0617c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff936b34638800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b0617c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b34638800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.249321 s, 67.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384652 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560403 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742557 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782539 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901155 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846805 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836269 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078261 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986929 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113104 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122131 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113827 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227129 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239552 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26875 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282739 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015006 s, 69.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203627 s, 51.5 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205523 s, 51.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190543 s, 55.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203412 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170165 s, 61.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021655 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0170588 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143875 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204271 s, 51.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211667 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400173 s, 262 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244959 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258019 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00473024 s, 222 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00558075 s, 188 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96394 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96512 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291775 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054803 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552949 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505076 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535023 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754386 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742706 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759738 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990218 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910041 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891641 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106921 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115036 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212321 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211141 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284019 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272073 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175534 s, 59.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174114 s, 60.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021078 s, 49.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222112 s, 47.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178827 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230078 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248878 s, 42.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302383 s, 34.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271208 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00715586 s, 147 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213986 s, 49.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00434091 s, 242 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00440784 s, 238 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00421565 s, 249 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296902 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00614595 s, 171 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg335-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg335-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.95457 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.76955 s, 10.7 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (116s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 18:52:27 (1779317547) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg335-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (9s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 18:52:35 (1779317555) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 18:52:39 (1779317559) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg335-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.40343 s, 308 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg335-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0159 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 18:53:30 (1779317610) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151678 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0446549 s, 23.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336977 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044785 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663905 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767392 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778213 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858797 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866458 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123531 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988132 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148681 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139202 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150005 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143981 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153465 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149234 s, 7.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff936b0617c000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff936b0617c000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249559 s, 42.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220061 s, 47.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266434 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203184 s, 51.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202629 s, 51.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268008 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232362 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284151 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288331 s, 36.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03113 s, 33.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295059 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320304 s, 32.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279576 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287231 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357497 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033352 s, 31.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.96774 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97408 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.146978 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453872 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951488 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112751 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135359 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1222 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130319 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127399 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134358 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123894 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14814 s, 7.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141687 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156809 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199197 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194185 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251598 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246488 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023186 s, 45.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221169 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0225354 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022492 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248588 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241385 s, 43.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220168 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023695 s, 44.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206841 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255328 s, 41.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257661 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232381 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249887 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248478 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280587 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285031 s, 36.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg335-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg335-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9362 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98006 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (70s) == sanityn test 77o: Changing rank should not panic ====== 18:54:40 (1779317680) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 18:54:45 (1779317685) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (100s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 18:56:26 (1779317786) mds.MDS.mdt.nrs_policies=tbf oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg335-client: oleg335-server: ssh exited with exit code 36 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg335-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg335-client: oleg335-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (21s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 18:56:46 (1779317806) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 271468 kill md_thread xargs PASS 77r (46s) == sanityn test 77s: Check TBF LRU shrinker ============== 18:57:33 (1779317853) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (46s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 18:58:19 (1779317899) oleg335-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg335-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg335-client: oleg335-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (8s) == sanityn test 79: xattr: intent error ================== 18:58:27 (1779317907) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935718 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 18:58:28.000000000 -0400 Modify: 2026-05-20 18:58:28.000000000 -0400 Change: 2026-05-20 18:58:28.000000000 -0400 Birth: 2026-05-20 18:58:28.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (16s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 18:58:43 (1779317923) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 18:58:46 (1779317926) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 18:58:49 (1779317929) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 18:58:52 (1779317932) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (2s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 18:58:55 (1779317935) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 18:58:57 (1779317937) - open/close 718 (time 1779317952.24 total 10.00 last 71.80) - open/close 1419 (time 1779317962.25 total 20.01 last 70.04) - open/close 2125 (time 1779317972.25 total 30.02 last 70.55) total: 2500 open/close in 35.42 seconds: 70.59 ops/second snapshot_time 1779318056.008170096 secs.nsecs start_time 1779317940.119715571 secs.nsecs elapsed_time 115.888454525 secs.nsecs open 2505 samples [usecs] 154 18755 4089885 10668352705 close 2504 samples [usecs] 188 7730 1047473 945360665 mknod 2501 samples [usecs] 582 18645 3590515 8363587025 unlink 2500 samples [usecs] 724 53598 4875954 26679983094 mkdir 2 samples [usecs] 1259 3047 4306 10869290 rmdir 2 samples [usecs] 9028 11751 20779 219590785 rename 2500 samples [usecs] 628 40678 16777919 209289204701 getattr 3762 samples [usecs] 7 6632 225980 169816418 parallel_rename_file 2500 samples [usecs] 628 40678 16777919 209289204701 crossdir_rename 2500 samples [usecs] 628 40678 16777919 209289204701 PASS 81d (122s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 19:00:59 (1779318059) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7570 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7570 PASS 82 (5s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 19:01:04 (1779318064) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 19:01:07 (1779318067) fail_loc=0x8000060b PASS 84 (10s) == sanityn test 85: Lustre API root cache race =========== 19:01:17 (1779318077) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1779318078 Finishing test test1 at 1779318081 PASS 85 (7s) == sanityn test 90: open/create and unlink striped directory ========================================================== 19:01:24 (1779318084) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 19:01:27 (1779318087) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (2s) == sanityn test 92: create remote directory under orphan directory ========================================================== 19:01:29 (1779318089) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (2s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 19:01:31 (1779318091) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5845 0x16d5 0x240000400 1 5808 0x16b0 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5809 0x16b1 0x280000400 0 5846 0x16d6 0x240000400 rc1=0 and rc2=0 PASS 93 (12s) == sanityn test 94: signal vs CP callback race =========== 19:01:44 (1779318104) osc.lustre-OST0000-osc-ffff936b0617c000.idle_timeout=0 osc.lustre-OST0000-osc-ffff936b34638800.idle_timeout=0 osc.lustre-OST0001-osc-ffff936b0617c000.idle_timeout=0 osc.lustre-OST0001-osc-ffff936b34638800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.141792 s, 57.8 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.00632 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 288856 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0497653 s, 82.3 kB/s osc.lustre-OST0000-osc-ffff936b0617c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff936b34638800.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b0617c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff936b34638800.idle_timeout=debug PASS 94 (12s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 19:01:57 (1779318117) llite.lustre-ffff936b0617c000.fast_read=0 llite.lustre-ffff936b34638800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0358659 s, 58.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff936b0617c000.fast_read=1 llite.lustre-ffff936b34638800.fast_read=1 PASS 95a (17s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 19:02:14 (1779318134) llite.lustre-ffff936b0617c000.fast_read=0 llite.lustre-ffff936b34638800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00928464 s, 1.3 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff936b0617c000.fast_read=1 llite.lustre-ffff936b34638800.fast_read=1 PASS 95b (11s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 19:02:25 (1779318145) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (2s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 19:02:27 (1779318147) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00727026 s, 563 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (5s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 19:02:32 (1779318152) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124683 s, 16.8 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (5s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 19:02:37 (1779318157) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128642 s, 16.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (5s) == sanityn test 100e: DoM: read on open and file size ==== 19:02:42 (1779318162) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00614491 s, 2.0 MB/s PASS 100e (4s) == sanityn test 101a: Discard DoM data on unlink ========= 19:02:46 (1779318166) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00611822 s, 669 kB/s PASS 101a (5s) == sanityn test 101b: Discard DoM data on rename ========= 19:02:51 (1779318171) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00751264 s, 545 kB/s PASS 101b (4s) == sanityn test 101c: Discard DoM data on close-unlink === 19:02:55 (1779318175) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.005081 s, 806 kB/s PASS 101c (6s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 19:03:02 (1779318182) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1779318184 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1779318187 Finishing test test23 at 1779318187 PASS 103 (11s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 19:03:13 (1779318193) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 19:03:15 (1779318195) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 19:03:41 (1779318221) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (3s) == sanityn test 106b: Glimpse RPCs test for statx ======== 19:03:43 (1779318223) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0219642 s, 46.6 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938253 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 19:03:44.000000000 -0400 Modify: 2026-05-20 19:03:44.000000000 -0400 Change: 2026-05-20 19:03:44.000000000 -0400 Birth: 2026-05-20 19:03:44.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938253 -rw-r--r-- 1 0 0 1779318224 1779318224 1779318224 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938253 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 19:03:44.000000000 -0400 Modify: 2026-05-20 19:03:44.000000000 -0400 Change: 2026-05-20 19:03:44.000000000 -0400 Birth: 2026-05-20 19:03:44.000000000 -0400 1779318224 1024 11 PASS 106b (6s) == sanityn test 106c: Verify statx attributes mask ======= 19:03:49 (1779318229) PASS 106c (5s) == sanityn test 107a: Basic grouplock conflict =========== 19:03:54 (1779318234) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.223384 s, 46.9 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5854 0x16de 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 First grouplock blocks second one PASS 107a (6s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 19:04:00 (1779318240) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.200992 s, 52.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5816 0x16b8 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7570 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (11s) == sanityn test 108a: lseek: parallel updates ============ 19:04:11 (1779318251) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00460111 s, 2.2 MB/s llite.lustre-ffff936b0617c000.hybrid_io=0 llite.lustre-ffff936b34638800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.41794 s, 1.9 MB/s PASS 108a (9s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 19:04:20 (1779318260) 192.168.203.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre (opts:) Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre 192.168.203.135@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.135@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg335-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg335-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre2 PASS 109 (39s) == sanityn test 110: do not grant another lock on resend ========================================================== 19:04:59 (1779318299) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-20 19:05:00.000000000 -0400 Modify: 2026-05-20 19:05:00.000000000 -0400 Change: 2026-05-20 19:05:00.000000000 -0400 Birth: 2026-05-20 19:05:00.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg335-client.virtnet oleg335-client.virtnet PASS 110 (166s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 19:07:45 (1779318465) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (5s) == sanityn test 112: update max-inherit in default LMV === 19:07:51 (1779318471) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (7s) == sanityn test 113: check servers of specified fs ======= 19:07:57 (1779318477) lustre-OST0000-osc-ffff936b07642800 active. lustre-OST0001-osc-ffff936b07642800 active. lustre-OST0000-osc-ffff936b07642800 active. lustre-OST0001-osc-ffff936b07642800 active. MGC192.168.203.135@tcp active. PASS 113 (9s) == sanityn test 114: implicit default LMV inherit ======== 19:08:06 (1779318486) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (5s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 19:08:11 (1779318491) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 19:08:15 (1779318495) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (5s) == sanityn test 121: trunc append race =================== 19:08:20 (1779318500) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0241418 s, 848 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.129609 s, 158 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (13s) == sanityn test 200: service remains healthy while able to process request ========================================================== 19:08:33 (1779318513) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=8 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2048 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1611 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2993 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2186 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1692 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2519 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1978 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.304 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2091 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2811 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.345 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4008 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3742 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4901 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3214 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5439 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg335-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff936b07642800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff936b07642800.ost_server_uuid in FULL state after 0 sec PASS 200 (85s) cleanup: ====================================================== == sanityn test complete, duration 8024 sec ============== 19:10:00 (1779318600) === sanityn: start cleanup 19:10:02 (1779318602) === Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg335-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 19:15:19 (1779318919) ===