-----============= acceptance-small: sanityn ============----- Tue Jun 16 01:28:00 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg644-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 01:28:21 (1781587701) === Starting client oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 Mount client oleg644-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 Started clients oleg644-client.virtnet: 192.168.206.144@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg644-client.virtnet: executing check_config_client /mnt/lustre oleg644-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg644-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93d0c2feb000.idle_timeout=debug osc.lustre-OST0000-osc-ffff93d0c5589800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0c2feb000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0c5589800.idle_timeout=debug disable quota as required oleg644-server: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 01:28:54 (1781587734) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307234 s, 34.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7596] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 01:28:56 (1781587736) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 01:29:04 (1781587744) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 01:29:11 (1781587751) -rw-r--r-- 1 root root 0 Jun 16 01:29 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (8s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 01:29:19 (1781587759) -rw-r--r-- 1 root root 0 Jun 16 01:29 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 01:29:27 (1781587767) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 01:29:34 (1781587774) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 01:29:41 (1781587781) total 1 -rwxrwxrwx 1 root root 0 Jun 16 01:29 f2a -rwxrwxrwx 1 root root 0 Jun 16 01:29 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 16 01:29 f2a -rwxrwxrwx 1 root root 0 Jun 16 01:29 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0193925 s, 26.4 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 01:29:48 (1781587788) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 01:29:52 (1781587792) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.330279 s, 6.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (8s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 01:30:00 (1781587800) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 01:30:07 (1781587807) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 01:30:16 (1781587816) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 01:30:23 (1781587823) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 01:30:31 (1781587831) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 01:30:39 (1781587839) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 01:30:46 (1781587846) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 01:30:54 (1781587854) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687811 s, 15.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0737898 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 01:31:01 (1781587861) 1+0 records in 1+0 records out 1 byte copied, 0.0120404 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0158732 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0211386 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0425196 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201746 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0224751 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0260342 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0249553 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0131913 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0227657 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0279581 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0150199 s, 0.1 kB/s PASS 10a (10s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 01:31:11 (1781587871) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.026652 s, 115 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0587028 s, 69.8 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00267112 s, 1.2 MB/s PASS 10b (9s) == sanityn test 11: execution of file opened for write should return error ============================================================== 01:31:20 (1781587880) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 01:31:28 (1781587888) warning: '-runtime' deprecated, use '-t runtime' instead - link 328 (time 1781587900.95 total 10.00 last 32.80) - link 617 (time 1781587910.96 total 20.02 last 28.85) - link 867 (time 1781587921.01 total 30.06 last 24.90) - link 1171 (time 1781587931.01 total 40.06 last 30.39) - link 1493 (time 1781587941.02 total 50.07 last 32.16) - link 1819 (time 1781587951.03 total 60.08 last 32.58) - link 2157 (time 1781587961.05 total 70.11 last 33.71) - link 2469 (time 1781587971.06 total 80.12 last 31.17) - link 2777 (time 1781587981.08 total 90.14 last 30.73) total: 3105 link in 100.02 seconds: 31.04 ops/second using seed 1183235062 running for 100 seconds - stat 4002 (time 1781587903 ; total 11 ; last 11) - stat 8166 (time 1781587914 ; total 22 ; last 11) - stat 10000 (time 1781587919 ; total 27 ; last 5) - stat 14618 (time 1781587930 ; total 38 ; last 11) - stat 19513 (time 1781587941 ; total 49 ; last 11) - stat 20000 (time 1781587942 ; total 50 ; last 1) - stat 22106 (time 1781587953 ; total 61 ; last 11) - stat 24329 (time 1781587964 ; total 72 ; last 11) - stat 26465 (time 1781587975 ; total 83 ; last 11) - stat 28716 (time 1781587986 ; total 94 ; last 11) - stat 30000 (time 1781587991 ; total 99 ; last 5) total: 30082 stats in 100 seconds: 300.820007 stats/second - unlinked 0 (time 1781587942 ; total 1 ; last 1) unlink(/mnt/lustre2/lockdir/lockfile3105) error: No such file or directory total: 3104 unlinks in 87 seconds: 35.678162 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21406) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21408) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 01:33:54 (1781588034) f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 01:34:03 (1781588043) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 22593 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 01:34:11 (1781588051) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 23172 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (9s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 01:34:20 (1781588060) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23751 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 01:34:28 (1781588068) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24334 Terminated $DIR1/$tdir/sleep 60 PASS 14c (8s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 01:34:36 (1781588076) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24916 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 01:34:45 (1781588085) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (49s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 01:35:34 (1781588134) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.491096 s, 17.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.327537 s, 25.6 MB/s Chance of close/open is 1 in 50 Seed set to 8137 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1781588138.417563 mapwrite 0x2e6263 thru 0x2ec503 (0x62a1 bytes) 000003[0] 1781588138.571420 read 0x0e3797 thru 0x0e60ef (0x2959 bytes) 000004[0] 1781588138.603271 mapwrite 0x768d24 thru 0x772879 (0x9b56 bytes) 000006[0] 1781588138.747422 write 0x14b723 thru 0x15b650 (0xff2e bytes) 000009[0] 1781588138.813369 read 0x6f2eea thru 0x702b3c (0xfc53 bytes) truncating to largest ever: 0x5f0336 000017[0] 1781588138.884132 trunc from 0x77287a to 0x5f0335 (0x182544 bytes) 000019[1] 1781588138.934352 mapwrite 0x8a8e2b thru 0x8b7add (0xecb3 bytes) 000020[0] 1781588139.128050 write 0x0bf2bd thru 0x0c9dc9 (0xab0d bytes) truncating to largest ever: 0x68cb76 000024[0] 1781588139.229122 trunc from 0x8b7ade to 0x68cb75 (0x22af68 bytes) 000026[0] 1781588139.287294 trunc from 0x68cb76 to 0x5c6632 (0xc6543 bytes) 000029[0] 1781588139.344248 mapread 0x37dedf thru 0x37ec45 (0xd67 bytes) 000032[0] 1781588139.444350 read 0x57f584 thru 0x583e7a (0x48f7 bytes) 000034[0] 1781588139.462134 write 0x2b1c17 thru 0x2bf0af (0xd499 bytes) 000035[1] 1781588139.496940 read 0x38923e thru 0x38bf4b (0x2d0e bytes) 000036[0] 1781588139.541345 mapwrite 0x8b2d7a thru 0x8bc5eb (0x9872 bytes) 000040[0] 1781588139.841553 mapwrite 0x895fe8 thru 0x89917e (0x3197 bytes) 000045[1] 1781588140.046755 trunc from 0x8bc5ec to 0x685288 (0x237363 bytes) 000046[1] 1781588140.196999 trunc from 0x685289 to 0x3f51d4 (0x2900b4 bytes) 000047[0] 1781588140.223315 write 0x4b3bee thru 0x4c3877 (0xfc8a bytes) 000049[0] 1781588140.292473 mapread 0x26fc01 thru 0x278bab (0x8fab bytes) 000051[1] 1781588140.380716 mapread 0x0f0096 thru 0x0f7a48 (0x79b3 bytes) 000052[0] 1781588140.473964 write 0x88b84c thru 0x8954fb (0x9cb0 bytes) 000056[0] 1781588140.588068 mapwrite 0x669847 thru 0x66b115 (0x18cf bytes) 000058[0] 1781588140.715175 read 0x220f7e thru 0x22659e (0x5621 bytes) 000059[1] 1781588140.743474 read 0x60c5db thru 0x61b7b0 (0xf1d6 bytes) 000060[0] 1781588140.777275 mapread 0x287ae2 thru 0x29679b (0xecba bytes) truncating to largest ever: 0x6b2119 000061[0] 1781588140.841635 trunc from 0x8954fc to 0x6b2118 (0x1e33e3 bytes) 000063[1] 1781588141.018171 read 0x16100d thru 0x165416 (0x440a bytes) 000065[1] 1781588141.146288 write 0x4117aa thru 0x419d75 (0x85cc bytes) 000066[0] 1781588141.176300 read 0x1663a6 thru 0x1759c1 (0xf61c bytes) 000067[0] 1781588141.224625 write 0x27d904 thru 0x286aa1 (0x919e bytes) 000068[1] 1781588141.284106 write 0x9c6623 thru 0x9cb6aa (0x5088 bytes) 000069[1] 1781588141.320558 trunc from 0x9cb6ab to 0x56dd29 (0x45d981 bytes) 000071[0] 1781588141.347148 write 0x5e6f11 thru 0x5f058e (0x967e bytes) 000073[1] 1781588141.426938 mapwrite 0x2215b9 thru 0x225349 (0x3d91 bytes) 000076[1] 1781588141.620501 mapread 0x4c3ed0 thru 0x4d23a9 (0xe4da bytes) 000077[1] 1781588141.704609 read 0x04cd74 thru 0x04f011 (0x229e bytes) 000079[0] 1781588141.727377 mapwrite 0x285a80 thru 0x286853 (0xdd4 bytes) 000082[1] 1781588141.852658 trunc from 0x5f058f to 0x0cc8e2 (0x523cac bytes) 000084[1] 1781588141.936187 trunc from 0x0cc8e3 to 0x528311 (0x45ba2f bytes) 000085[1] 1781588141.968332 read 0x263a39 thru 0x266235 (0x27fd bytes) 000086[1] 1781588141.980172 write 0x3b72e0 thru 0x3bedfc (0x7b1d bytes) 000087[1] 1781588142.017692 mapwrite 0x86228a thru 0x86ce56 (0xabcd bytes) 000091[1] 1781588142.170376 mapread 0x70398c thru 0x70ea1c (0xb091 bytes) truncating to largest ever: 0x7b4d88 000093[1] 1781588142.230663 trunc from 0x86ce57 to 0x7b4d87 (0xb80cf bytes) 000094[1] 1781588142.259297 trunc from 0x7b4d88 to 0x082483 (0x732904 bytes) 000095[0] 1781588142.278195 read 0x06eeae thru 0x07ea68 (0xfbbb bytes) 000096[0] 1781588142.312310 read 0x01651c thru 0x020d0a (0xa7ef bytes) 000097[1] 1781588142.335282 read 0x049701 thru 0x04f790 (0x6090 bytes) 000098[0] 1781588142.425328 mapwrite 0x2254a2 thru 0x22ba9e (0x65fd bytes) 000101[0] 1781588142.616967 mapread 0x2018b1 thru 0x204983 (0x30d3 bytes) 000106[1] 1781588142.669331 mapread 0x093e7d thru 0x0a33fb (0xf57f bytes) 000107[0] 1781588142.779177 mapwrite 0x382ae0 thru 0x388753 (0x5c74 bytes) 000109[1] 1781588143.000166 trunc from 0x388754 to 0x3efbf0 (0x6749d bytes) 000110[1] 1781588143.076363 write 0x823eb4 thru 0x830594 (0xc6e1 bytes) 000111[1] 1781588143.083672 write 0x07278d thru 0x073133 (0x9a7 bytes) 000113[0] 1781588143.186276 write 0x2ba54a thru 0x2c8960 (0xe417 bytes) truncating to largest ever: 0x7e0723 000114[0] 1781588143.249941 trunc from 0x830595 to 0x7e0722 (0x4fe72 bytes) 000116[1] 1781588143.340982 mapwrite 0x962622 thru 0x967522 (0x4f01 bytes) 000117[0] 1781588143.714255 mapread 0x29be2a thru 0x2a2fbe (0x7195 bytes) 000118[1] 1781588143.805673 read 0x45a3cb thru 0x4656f3 (0xb329 bytes) 000121[0] 1781588143.807524 mapread 0x60d001 thru 0x611106 (0x4106 bytes) 000122[1] 1781588143.934446 mapread 0x39bdec thru 0x3a0979 (0x4b8e bytes) 000127[1] 1781588143.972206 trunc from 0x967523 to 0x6b9fed (0x2ad535 bytes) 000130[1] 1781588144.131796 mapread 0x0baa44 thru 0x0be492 (0x3a4f bytes) 000133[0] 1781588144.187596 trunc from 0x6b9fee to 0x5b6f7e (0x10306f bytes) 000134[0] 1781588144.236157 trunc from 0x5b6f7f to 0x61d420 (0x664a2 bytes) truncating to largest ever: 0x9d1666 000140[0] 1781588144.360014 trunc from 0x61d421 to 0x9d1665 (0x3b4245 bytes) 000141[1] 1781588144.430199 read 0x1a8a67 thru 0x1b1b99 (0x9133 bytes) 000142[1] 1781588144.466309 mapread 0x74845f thru 0x74fda3 (0x7945 bytes) 000144[1] 1781588144.504844 mapread 0x9cbfff thru 0x9d1665 (0x5667 bytes) 000145[0] 1781588144.555408 write 0x99d2cf thru 0x9a41ce (0x6f00 bytes) 000148[1] 1781588144.692174 write 0x30303c thru 0x30d787 (0xa74c bytes) 000150[0] 1781588144.789543 mapread 0x6697e7 thru 0x66a470 (0xc8a bytes) 000151[0] 1781588144.827173 write 0x130636 thru 0x139a73 (0x943e bytes) 000158[0] 1781588144.992832 mapwrite 0x9ba6f4 thru 0x9bb2aa (0xbb7 bytes) 000159[1] 1781588145.174175 trunc from 0x9d1666 to 0x6bd196 (0x3144cf bytes) 000160[0] 1781588145.228261 mapwrite 0x96d6bc thru 0x97bb51 (0xe496 bytes) 000164[0] 1781588145.537682 read 0x49c964 thru 0x4a0594 (0x3c31 bytes) 000166[1] 1781588145.560641 mapread 0x11596c thru 0x11e4a8 (0x8b3d bytes) 000167[1] 1781588145.648642 mapwrite 0x27b50c thru 0x27bb90 (0x685 bytes) 000169[0] 1781588145.832779 read 0x32478b thru 0x32e3f0 (0x9c66 bytes) 000170[0] 1781588145.914332 trunc from 0x97bb52 to 0x8b368d (0xc84c4 bytes) 000171[1] 1781588146.024706 read 0x6d72f8 thru 0x6d7688 (0x391 bytes) 000173[1] 1781588146.075917 mapread 0x584357 thru 0x58709f (0x2d49 bytes) 000174[0] 1781588146.112366 mapread 0x16eaab thru 0x1778c4 (0x8e1a bytes) 000175[0] 1781588146.135779 trunc from 0x8b368e to 0x1466ec (0x76cfa1 bytes) 000176[0] 1781588146.207233 read 0x06d52d thru 0x075fe1 (0x8ab5 bytes) 000185[1] 1781588146.369307 read 0x0b5733 thru 0x0b9180 (0x3a4e bytes) 000187[1] 1781588146.451395 read 0x02cc9b thru 0x03beec (0xf252 bytes) 000188[1] 1781588146.457553 trunc from 0x1466ed to 0x53f7ec (0x3f9100 bytes) 000189[0] 1781588146.531538 trunc from 0x53f7ed to 0x792c75 (0x253489 bytes) 000190[0] 1781588146.608708 mapwrite 0x07edf8 thru 0x087d9c (0x8fa5 bytes) 000191[1] 1781588146.870073 mapread 0x776f4a thru 0x780037 (0x90ee bytes) 000193[1] 1781588147.102247 write 0x359fea thru 0x35a937 (0x94e bytes) 000194[0] 1781588147.204403 write 0x10f013 thru 0x111af2 (0x2ae0 bytes) 000197[0] 1781588147.288496 write 0x7548b9 thru 0x75efdf (0xa727 bytes) 000199[0] 1781588147.480419 read 0x20843d thru 0x21804d (0xfc11 bytes) 000201[0] 1781588147.513706 mapwrite 0x297801 thru 0x29ff42 (0x8742 bytes) 000205[1] 1781588147.721249 mapread 0x44a164 thru 0x4580fc (0xdf99 bytes) 000206[0] 1781588147.905076 read 0x4087c0 thru 0x40d087 (0x48c8 bytes) 000207[1] 1781588147.952159 read 0x49bcd7 thru 0x4a8c34 (0xcf5e bytes) 000210[0] 1781588148.000103 mapwrite 0x150a9c thru 0x15a6c3 (0x9c28 bytes) 000212[0] 1781588148.173276 trunc from 0x792c76 to 0x1912ae (0x6019c7 bytes) 000215[0] 1781588148.265393 trunc from 0x1912af to 0x8bf9b0 (0x72e702 bytes) 000217[0] 1781588148.328159 write 0x51a79b thru 0x51aebf (0x725 bytes) 000218[0] 1781588148.377373 write 0x74b5e6 thru 0x75493a (0x9355 bytes) 000219[0] 1781588148.422536 trunc from 0x8bf9b1 to 0x44dcfb (0x471cb5 bytes) 000220[1] 1781588148.492761 mapread 0x1940b4 thru 0x1a2289 (0xe1d6 bytes) 000221[1] 1781588148.618228 trunc from 0x44dcfc to 0x0cdcd2 (0x380029 bytes) 000230[0] 1781588148.935896 write 0x92c4dc thru 0x93b781 (0xf2a6 bytes) 000231[0] 1781588148.983336 read 0x5f96da thru 0x5fefea (0x5911 bytes) 000232[0] 1781588149.020073 mapread 0x1b9bf8 thru 0x1bb1e3 (0x15ec bytes) 000235[0] 1781588149.052372 read 0x77fc5d thru 0x786ce9 (0x708d bytes) 000237[1] 1781588149.060239 trunc from 0x93b782 to 0x78d2e3 (0x1ae49e bytes) 000241[0] 1781588149.200546 mapread 0x22e518 thru 0x234d14 (0x67fd bytes) 000242[0] 1781588149.287588 mapwrite 0x9b9c62 thru 0x9bf9bc (0x5d5b bytes) 000244[1] 1781588149.448045 read 0x252d2f thru 0x262b65 (0xfe37 bytes) 000245[0] 1781588149.487083 read 0x18f826 thru 0x19e7ba (0xef95 bytes) 000248[0] 1781588149.526815 read 0x6e1170 thru 0x6e48ec (0x377d bytes) 000253[1] 1781588149.567380 mapread 0x7865cb thru 0x792a5d (0xc493 bytes) 000254[0] 1781588149.631009 read 0x429e2e thru 0x4330fc (0x92cf bytes) 000262[1] 1781588149.653708 read 0x8ea140 thru 0x8ee8b5 (0x4776 bytes) 000263[0] 1781588149.668995 mapwrite 0x1ce2ba thru 0x1db010 (0xcd57 bytes) 000265[1] 1781588149.787662 trunc from 0x9bf9bd to 0x6886bc (0x337300 bytes) 000267[0] 1781588149.841021 mapread 0x5e7866 thru 0x5efe34 (0x85cf bytes) 000269[0] 1781588149.947992 trunc from 0x6886bd to 0x6d1148 (0x48a8c bytes) 000271[0] 1781588150.073312 mapread 0x2abae3 thru 0x2ba2fb (0xe819 bytes) 000280[1] 1781588150.283816 mapread 0x35ae80 thru 0x363ddb (0x8f5c bytes) 000281[0] 1781588150.383111 mapread 0x5eebb0 thru 0x5f6af4 (0x7f45 bytes) 000282[0] 1781588150.429735 mapread 0x4d82b8 thru 0x4d98f3 (0x163c bytes) 000283[1] 1781588150.454734 trunc from 0x6d1149 to 0x4a0e2d (0x23031b bytes) 000287[1] 1781588150.537112 trunc from 0x4a0e2e to 0x0b86f7 (0x3e8736 bytes) 000289[0] 1781588150.584790 mapread 0x06e4a7 thru 0x0753c3 (0x6f1d bytes) 000292[0] 1781588150.651373 read 0x05125a thru 0x051646 (0x3ed bytes) 000293[0] 1781588150.672225 mapread 0x07ecc7 thru 0x082ded (0x4127 bytes) 000294[1] 1781588150.682884 mapread 0x06b541 thru 0x076799 (0xb259 bytes) 000295[1] 1781588150.738687 mapread 0x09a0a4 thru 0x09f8ef (0x584c bytes) 000299[0] 1781588150.807819 mapwrite 0x958cff thru 0x95fc5f (0x6f61 bytes) 000300[1] 1781588151.102230 mapwrite 0x94c89f thru 0x953a75 (0x71d7 bytes) 000304[1] 1781588151.309208 mapread 0x882e81 thru 0x88817a (0x52fa bytes) 000305[0] 1781588151.363540 mapwrite 0x4b6108 thru 0x4c4879 (0xe772 bytes) 000306[0] 1781588151.492387 mapread 0x8a0c7f thru 0x8a6214 (0x5596 bytes) 000307[1] 1781588151.635591 read 0x1a955f thru 0x1b2b0a (0x95ac bytes) 000308[0] 1781588151.675156 write 0x19600e thru 0x1a5466 (0xf459 bytes) 000310[0] 1781588151.723536 read 0x12f11c thru 0x13d8d2 (0xe7b7 bytes) 000313[0] 1781588151.756120 trunc from 0x95fc60 to 0x2c4f5c (0x69ad03 bytes) 000314[1] 1781588151.873401 mapwrite 0x8e28f4 thru 0x8e5ae8 (0x31f5 bytes) 000318[1] 1781588152.127989 write 0x4bef88 thru 0x4c6aa4 (0x7b1d bytes) 000319[1] 1781588152.149073 read 0x1ed9ca thru 0x1f814c (0xa783 bytes) 000320[0] 1781588152.164295 read 0x22dad6 thru 0x2366c6 (0x8bf1 bytes) 000322[0] 1781588152.231609 read 0x520ce1 thru 0x5213c6 (0x6e6 bytes) 000326[1] 1781588152.277747 trunc from 0x8e5ae9 to 0x8d23aa (0x1373e bytes) 000327[0] 1781588152.380643 trunc from 0x8d23ab to 0x3c759c (0x50ae0e bytes) 000328[0] 1781588152.473930 write 0x6dda22 thru 0x6e991d (0xbefc bytes) 000329[0] 1781588152.483386 write 0x08a325 thru 0x099875 (0xf551 bytes) 000330[1] 1781588152.521233 write 0x11fa77 thru 0x11ff6a (0x4f4 bytes) 000331[1] 1781588152.576321 read 0x2a2066 thru 0x2ae88a (0xc825 bytes) 000332[1] 1781588152.611640 read 0x6d0519 thru 0x6dd96c (0xd454 bytes) 000334[0] 1781588152.967364 write 0x9e841b thru 0x9ee9d0 (0x65b6 bytes) 000335[1] 1781588153.004664 mapread 0x5e7615 thru 0x5e8af9 (0x14e5 bytes) 000337[1] 1781588153.038623 read 0x62d3cd thru 0x637386 (0x9fba bytes) 000341[0] 1781588153.108625 trunc from 0x9ee9d1 to 0x3e5685 (0x60934b bytes) 000342[0] 1781588153.260266 write 0x9bd659 thru 0x9caa91 (0xd439 bytes) 000344[0] 1781588153.272322 mapread 0x39a273 thru 0x3a4ad4 (0xa862 bytes) 000345[0] 1781588153.420300 mapwrite 0x42ca09 thru 0x434d5f (0x8357 bytes) 000348[1] 1781588153.653454 mapread 0x944b41 thru 0x94f740 (0xac00 bytes) 000351[1] 1781588153.761344 write 0x0df310 thru 0x0edfab (0xec9c bytes) 000352[1] 1781588153.821424 mapwrite 0x84ec96 thru 0x84efa6 (0x311 bytes) 000353[0] 1781588153.918665 mapwrite 0x51efdc thru 0x5239c5 (0x49ea bytes) 000354[0] 1781588154.140737 write 0x2dedb0 thru 0x2e4de3 (0x6034 bytes) 000356[1] 1781588154.189487 read 0x3fc54d thru 0x402779 (0x622d bytes) 000360[0] 1781588154.273221 read 0x229613 thru 0x22ba78 (0x2466 bytes) 000361[1] 1781588154.280844 mapread 0x10d27c thru 0x1185d0 (0xb355 bytes) 000362[0] 1781588154.343909 write 0x573d6b thru 0x5743e8 (0x67e bytes) 000364[0] 1781588154.521163 trunc from 0x9caa92 to 0x0f4a01 (0x8d6090 bytes) 000366[0] 1781588154.629594 write 0x1d7420 thru 0x1e701a (0xfbfb bytes) 000368[0] 1781588154.638754 read 0x0d33fe thru 0x0d6079 (0x2c7c bytes) 000369[0] 1781588154.692177 mapread 0x01bf11 thru 0x01ce80 (0xf70 bytes) 000371[1] 1781588154.751810 mapwrite 0x662aee thru 0x6718bc (0xedcf bytes) 000372[1] 1781588155.099104 read 0x262f6e thru 0x265ed6 (0x2f69 bytes) 000376[0] 1781588155.245279 mapread 0x54a770 thru 0x553e14 (0x96a5 bytes) 000378[1] 1781588155.442309 mapread 0x54792f thru 0x54daf6 (0x61c8 bytes) 000380[0] 1781588155.508512 write 0x2ede92 thru 0x2f7a6f (0x9bde bytes) 000381[0] 1781588155.625301 read 0x1d6ee6 thru 0x1de7d9 (0x78f4 bytes) 000383[0] 1781588155.653136 read 0x0db955 thru 0x0e00be (0x476a bytes) 000387[0] 1781588155.721558 mapread 0x4fa151 thru 0x509aab (0xf95b bytes) 000388[1] 1781588155.723087 trunc from 0x6718bd to 0x7a8d40 (0x137484 bytes) 000389[0] 1781588155.841471 read 0x15bec3 thru 0x16a7a3 (0xe8e1 bytes) 000390[1] 1781588155.883064 trunc from 0x7a8d41 to 0x0f21e5 (0x6b6b5b bytes) 000391[0] 1781588155.925386 write 0x6421ed thru 0x64898b (0x679f bytes) 000395[1] 1781588156.034348 write 0x932e18 thru 0x93a7d1 (0x79ba bytes) 000397[0] 1781588156.079092 read 0x465dc8 thru 0x46e36c (0x85a5 bytes) 000399[0] 1781588156.118824 mapwrite 0x006b3d thru 0x00a567 (0x3a2b bytes) 000400[0] 1781588156.286535 mapread 0x778c26 thru 0x785627 (0xca02 bytes) 000401[1] 1781588156.429231 trunc from 0x93a7d2 to 0x49a729 (0x4a00a8 bytes) 000402[0] 1781588156.501327 write 0x1ca34d thru 0x1d8c6d (0xe921 bytes) 000405[1] 1781588156.615363 mapwrite 0x78a8bd thru 0x7953c8 (0xab0c bytes) 000406[1] 1781588156.820533 mapwrite 0x84ac1f thru 0x85160d (0x69ef bytes) 000408[0] 1781588156.965773 mapread 0x38d35d thru 0x3909bf (0x3663 bytes) 000409[0] 1781588157.053489 trunc from 0x85160e to 0x8288c8 (0x28d45 bytes) 000411[1] 1781588157.197141 mapread 0x3fe980 thru 0x403114 (0x4795 bytes) 000412[1] 1781588157.329217 write 0x6d2b49 thru 0x6d56cf (0x2b87 bytes) 000413[0] 1781588157.381240 read 0x417f46 thru 0x420e3a (0x8ef5 bytes) 000414[0] 1781588157.450304 write 0x09ad98 thru 0x0a6990 (0xbbf9 bytes) 000415[1] 1781588157.549265 read 0x4e9eef thru 0x4f6fbb (0xd0cd bytes) 000416[1] 1781588157.564481 read 0x1d031e thru 0x1dd252 (0xcf35 bytes) 000418[0] 1781588157.627564 write 0x7dec21 thru 0x7e956f (0xa94f bytes) 000420[1] 1781588157.678651 mapread 0x3870ca thru 0x38f287 (0x81be bytes) 000421[0] 1781588157.718914 read 0x0012d0 thru 0x009701 (0x8432 bytes) 000423[1] 1781588157.750212 mapread 0x157c94 thru 0x1611a8 (0x9515 bytes) 000425[1] 1781588157.824417 mapread 0x7ebcec thru 0x7ecb13 (0xe28 bytes) 000426[0] 1781588157.872996 read 0x44c0c1 thru 0x44e8c5 (0x2805 bytes) 000428[1] 1781588157.887740 mapwrite 0x452937 thru 0x45ecc8 (0xc392 bytes) 000432[1] 1781588158.039137 read 0x3d4e8a thru 0x3e0012 (0xb189 bytes) 000433[0] 1781588158.106209 write 0x57d204 thru 0x584841 (0x763e bytes) 000434[1] 1781588158.140799 mapread 0x369880 thru 0x36d571 (0x3cf2 bytes) 000435[1] 1781588158.174054 write 0x9684ef thru 0x975b61 (0xd673 bytes) 000436[1] 1781588158.254768 trunc from 0x975b62 to 0x02e3b5 (0x9477ac bytes) 000439[0] 1781588158.454210 trunc from 0x02e3b6 to 0x3db044 (0x3acc8f bytes) 000443[0] 1781588158.600993 read 0x0d2cdf thru 0x0dcb03 (0x9e25 bytes) 000445[1] 1781588158.780278 mapread 0x2831fe thru 0x291ebd (0xecc0 bytes) 000446[1] 1781588158.892944 write 0x847aed thru 0x84a9ff (0x2f13 bytes) 000450[0] 1781588159.019699 trunc from 0x84aa00 to 0x4e84b3 (0x36254c bytes) 000456[1] 1781588159.205447 read 0x421e4c thru 0x42a289 (0x843e bytes) 000457[1] 1781588159.231372 write 0x16cb71 thru 0x17026e (0x36fe bytes) 000458[0] 1781588159.287506 write 0x4bce4b thru 0x4bf71c (0x28d2 bytes) 000459[1] 1781588159.346636 mapread 0x4ba2a2 thru 0x4c395c (0x96bb bytes) 000461[0] 1781588159.414895 mapread 0x2d1940 thru 0x2d23c7 (0xa88 bytes) 000463[0] 1781588159.478389 read 0x2c037d thru 0x2cf9dd (0xf661 bytes) 000464[1] 1781588159.494238 mapwrite 0x9bb0e7 thru 0x9c82b8 (0xd1d2 bytes) 000465[0] 1781588159.809007 write 0x0ed3cd thru 0x0f4324 (0x6f58 bytes) 000466[1] 1781588159.911773 mapread 0x11499a thru 0x119b51 (0x51b8 bytes) 000467[1] 1781588160.030610 read 0x8c3813 thru 0x8d08d1 (0xd0bf bytes) 000470[0] 1781588160.062711 mapwrite 0x3a9363 thru 0x3ac954 (0x35f2 bytes) 000474[1] 1781588160.295734 mapwrite 0x1c72cb thru 0x1ca2d2 (0x3008 bytes) 000475[0] 1781588160.486162 write 0x46a083 thru 0x4723c6 (0x8344 bytes) 000481[0] 1781588160.638092 read 0x6d6aa4 thru 0x6e5d70 (0xf2cd bytes) 000483[1] 1781588160.700064 read 0x5e0659 thru 0x5e5cd1 (0x5679 bytes) 000484[1] 1781588160.769363 read 0x3f7c7c thru 0x3fba5f (0x3de4 bytes) 000485[0] 1781588160.842126 trunc from 0x9c82b9 to 0x91b7a2 (0xacb16 bytes) 000486[1] 1781588160.994435 write 0x5167e0 thru 0x51c5c6 (0x5de7 bytes) 000489[1] 1781588161.114630 write 0x7b519e thru 0x7bfb84 (0xa9e7 bytes) 000492[1] 1781588161.194593 trunc from 0x91b7a3 to 0x72c3c3 (0x1ef3df bytes) 000493[1] 1781588161.236422 trunc from 0x72c3c4 to 0x17c19c (0x5b0227 bytes) 000500[1] 1781588161.358042 read 0x063b03 thru 0x063c35 (0x133 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 8161 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[1] 1781588162.406683 write 0x853000 thru 0x855042 (0x2043 bytes) 000013[0] 1781588162.586927 read_OD 0x2b5000 thru 0x2bcfff (0x8000 bytes) 000016[0] 1781588162.632077 write_OD 0x2a7000 thru 0x2b1fff (0xb000 bytes) 000017[1] 1781588162.706439 mapwrite 0x409000 thru 0x414974 (0xb975 bytes) 000020[1] 1781588162.829564 write 0x7d0000 thru 0x7dbf4e (0xbf4f bytes) 000023[0] 1781588162.846795 read_OD 0x31e000 thru 0x326fff (0x9000 bytes) 000024[1] 1781588162.917919 write 0x2dd000 thru 0x2e51e4 (0x81e5 bytes) 000025[1] 1781588162.965586 read 0x00f000 thru 0x013076 (0x4077 bytes) 000033[1] 1781588163.108459 read 0x095000 thru 0x0a38dc (0xe8dd bytes) truncating to largest ever: 0x158e17 000036[0] 1781588163.137692 trunc from 0x855043 to 0x158e16 (0x6fc22c bytes) 000037[0] 1781588163.204194 write_OD 0x15e000 thru 0x166fff (0x9000 bytes) 000038[1] 1781588163.263818 read 0x092000 thru 0x0a129d (0xf29e bytes) 000042[1] 1781588163.325420 mapread 0x02b000 thru 0x02ce56 (0x1e57 bytes) 000043[1] 1781588163.326524 write 0x30e000 thru 0x31c498 (0xe499 bytes) 000046[1] 1781588163.380374 read 0x140000 thru 0x141503 (0x1504 bytes) 000048[0] 1781588163.528458 write_OD 0x48c000 thru 0x492fff (0x7000 bytes) 000049[0] 1781588163.621522 mapwrite 0x3a2000 thru 0x3ad8a1 (0xb8a2 bytes) 000051[1] 1781588163.821233 mapread 0x307000 thru 0x312d77 (0xbd78 bytes) 000052[0] 1781588164.018617 mapread 0x0ba000 thru 0x0c1f76 (0x7f77 bytes) 000054[1] 1781588164.045088 read 0x2cf000 thru 0x2dcbe4 (0xdbe5 bytes) 000055[1] 1781588164.066268 mapread 0x40e000 thru 0x41105a (0x305b bytes) 000057[0] 1781588164.091096 mapwrite 0x9e7000 thru 0x9ee185 (0x7186 bytes) 000059[1] 1781588164.207628 mapread 0x5a8000 thru 0x5b7471 (0xf472 bytes) 000067[0] 1781588164.270654 mapread 0x927000 thru 0x92eea9 (0x7eaa bytes) 000068[1] 1781588164.284294 write 0x319000 thru 0x325efa (0xcefb bytes) 000071[0] 1781588164.339535 mapwrite 0x918000 thru 0x91ef33 (0x6f34 bytes) truncating to largest ever: 0x18c8af 000073[1] 1781588164.439014 trunc from 0x9ee186 to 0x18c8ae (0x8618d7 bytes) 000078[0] 1781588164.521139 mapread 0x013000 thru 0x01f405 (0xc406 bytes) 000079[1] 1781588164.628349 write 0x320000 thru 0x323e2a (0x3e2b bytes) 000080[1] 1781588164.665584 write 0x229000 thru 0x22eb2e (0x5b2f bytes) 000082[1] 1781588164.676162 read 0x01a000 thru 0x025fc5 (0xbfc6 bytes) 000085[1] 1781588164.757281 write 0x68b000 thru 0x68c6c6 (0x16c7 bytes) 000086[1] 1781588164.760254 mapread 0x2f6000 thru 0x2fffd1 (0x9fd2 bytes) 000091[0] 1781588164.847315 mapwrite 0x82f000 thru 0x8371ef (0x81f0 bytes) 000092[0] 1781588165.008256 read_OD 0x3b0000 thru 0x3bcfff (0xd000 bytes) 000094[1] 1781588165.032155 write_OD 0x038000 thru 0x041fff (0xa000 bytes) 000096[1] 1781588165.090360 mapread 0x179000 thru 0x1882e7 (0xf2e8 bytes) 000098[1] 1781588165.118984 mapread 0x57c000 thru 0x57fcd3 (0x3cd4 bytes) 000103[1] 1781588165.147692 mapread 0x773000 thru 0x779f35 (0x6f36 bytes) 000104[0] 1781588165.168138 mapwrite 0x6de000 thru 0x6e8086 (0xa087 bytes) 000107[1] 1781588165.277407 mapread 0x606000 thru 0x609d71 (0x3d72 bytes) 000109[0] 1781588165.330834 mapread 0x1f3000 thru 0x1ff2a7 (0xc2a8 bytes) 000112[0] 1781588165.391174 read_OD 0x5e6000 thru 0x5e7fff (0x2000 bytes) 000113[1] 1781588165.397509 write_OD 0x63f000 thru 0x64afff (0xc000 bytes) truncating to largest ever: 0x7e0ea2 000114[1] 1781588165.437928 trunc from 0x8371f0 to 0x7e0ea1 (0x5634e bytes) 000116[0] 1781588165.524161 mapwrite 0x7fd000 thru 0x7ff944 (0x2945 bytes) 000120[1] 1781588165.674842 mapwrite 0x30b000 thru 0x313e4d (0x8e4e bytes) 000121[0] 1781588165.790749 mapread 0x725000 thru 0x725d38 (0xd39 bytes) truncating to largest ever: 0x99c4b7 000122[0] 1781588165.838560 trunc from 0x7ff945 to 0x99c4b6 (0x19cb72 bytes) 000125[1] 1781588165.947546 mapwrite 0x654000 thru 0x660cbf (0xccc0 bytes) 000127[1] 1781588166.112275 write_OD 0x872000 thru 0x87ffff (0xe000 bytes) 000128[1] 1781588166.199953 read_OD 0x57f000 thru 0x589fff (0xb000 bytes) 000129[1] 1781588166.230024 read_OD 0x2b6000 thru 0x2b9fff (0x4000 bytes) 000133[1] 1781588166.245036 trunc from 0x99c4b7 to 0x23e8b1 (0x75dc05 bytes) 000134[1] 1781588166.290198 mapread 0x0ff000 thru 0x10d37e (0xe37f bytes) 000135[0] 1781588166.381070 trunc from 0x23e8b2 to 0x4172b8 (0x1d8a07 bytes) 000136[0] 1781588166.425458 trunc from 0x4172b9 to 0x1658dd (0x2b19db bytes) 000139[1] 1781588166.488959 write_OD 0x0f6000 thru 0x0f6fff (0x1000 bytes) 000140[1] 1781588166.557508 mapread 0x044000 thru 0x05334f (0xf350 bytes) 000141[0] 1781588166.578737 read_OD 0x0ee000 thru 0x0f1fff (0x4000 bytes) 000142[1] 1781588166.584718 write_OD 0x311000 thru 0x316fff (0x6000 bytes) 000143[1] 1781588166.649606 write_OD 0x6c1000 thru 0x6c3fff (0x3000 bytes) 000144[1] 1781588166.719140 read_OD 0x223000 thru 0x223fff (0x1000 bytes) 000145[1] 1781588166.733059 read_OD 0x62f000 thru 0x63afff (0xc000 bytes) 000146[0] 1781588166.743813 mapwrite 0x669000 thru 0x677710 (0xe711 bytes) 000147[1] 1781588166.890543 read_OD 0x4fc000 thru 0x505fff (0xa000 bytes) 000148[1] 1781588166.959631 write_OD 0x5d7000 thru 0x5dcfff (0x6000 bytes) 000149[1] 1781588167.031323 mapread 0x569000 thru 0x56f14b (0x614c bytes) 000150[1] 1781588167.070175 read_OD 0x2a1000 thru 0x2adfff (0xd000 bytes) 000152[1] 1781588167.092695 trunc from 0x6c4000 to 0x0ae120 (0x615edf bytes) 000154[0] 1781588167.163319 read_OD 0x06e000 thru 0x078fff (0xb000 bytes) 000155[1] 1781588167.190821 mapread 0x092000 thru 0x095c85 (0x3c86 bytes) 000158[0] 1781588167.217278 mapwrite 0x118000 thru 0x126bd2 (0xebd3 bytes) 000159[1] 1781588167.341471 trunc from 0x126bd3 to 0x40f24b (0x2e8679 bytes) 000161[1] 1781588167.383166 read_OD 0x2b0000 thru 0x2bafff (0xb000 bytes) 000162[0] 1781588167.421601 mapwrite 0x1a4000 thru 0x1a865f (0x4660 bytes) 000166[1] 1781588167.534569 write_OD 0x208000 thru 0x20bfff (0x4000 bytes) 000167[0] 1781588167.587128 read_OD 0x1c6000 thru 0x1ccfff (0x7000 bytes) 000168[1] 1781588167.606455 write_OD 0x51c000 thru 0x520fff (0x5000 bytes) 000169[1] 1781588167.669340 trunc from 0x521000 to 0x8a90eb (0x3880ec bytes) 000171[1] 1781588167.707035 mapwrite 0x934000 thru 0x93f7d2 (0xb7d3 bytes) 000174[0] 1781588167.900564 trunc from 0x93f7d3 to 0x2bee67 (0x68096b bytes) 000175[1] 1781588167.942725 mapwrite 0x9dc000 thru 0x9ddf09 (0x1f0a bytes) 000176[1] 1781588168.123241 write_OD 0x51a000 thru 0x51afff (0x1000 bytes) 000177[1] 1781588168.179170 mapread 0x77d000 thru 0x781fec (0x4fed bytes) 000181[0] 1781588168.219518 read_OD 0x1ae000 thru 0x1b7fff (0xa000 bytes) 000182[0] 1781588168.227401 mapread 0x607000 thru 0x610641 (0x9642 bytes) 000184[1] 1781588168.272527 trunc from 0x9ddf0a to 0x3bf7c0 (0x61e749 bytes) 000186[1] 1781588168.318756 mapread 0x1e6000 thru 0x1eca7a (0x6a7b bytes) 000188[0] 1781588168.341542 trunc from 0x3bf7c1 to 0x1b5246 (0x20a57a bytes) 000190[1] 1781588168.389758 read_OD 0x075000 thru 0x077fff (0x3000 bytes) 000191[0] 1781588168.432731 write_OD 0x774000 thru 0x779fff (0x6000 bytes) 000203[0] 1781588168.512318 trunc from 0x77a000 to 0x041866 (0x738799 bytes) 000204[0] 1781588168.564084 mapwrite 0x4f8000 thru 0x4fcf05 (0x4f06 bytes) 000206[1] 1781588168.722734 write_OD 0x4c8000 thru 0x4ccfff (0x5000 bytes) 000207[0] 1781588168.775473 mapread 0x2e8000 thru 0x2e8831 (0x832 bytes) 000209[0] 1781588168.794423 trunc from 0x4fcf06 to 0x4e5000 (0x17f05 bytes) 000210[1] 1781588168.874441 trunc from 0x4e5001 to 0x11d935 (0x3c76cb bytes) 000213[0] 1781588168.920938 write_OD 0x864000 thru 0x871fff (0xe000 bytes) 000217[1] 1781588168.974554 write_OD 0x4b2000 thru 0x4b5fff (0x4000 bytes) 000220[0] 1781588169.036712 mapwrite 0x656000 thru 0x66557a (0xf57b bytes) 000221[1] 1781588169.133168 mapwrite 0x148000 thru 0x1559cd (0xd9ce bytes) 000222[1] 1781588169.322657 mapwrite 0x4ce000 thru 0x4d3be8 (0x5be9 bytes) 000223[1] 1781588169.492927 read_OD 0x816000 thru 0x817fff (0x2000 bytes) 000224[1] 1781588169.522396 trunc from 0x872000 to 0x8814fd (0xf4fe bytes) 000226[1] 1781588169.580541 mapread 0x84e000 thru 0x84ed5f (0xd60 bytes) 000231[1] 1781588169.606239 read_OD 0x5f2000 thru 0x5f8fff (0x7000 bytes) 000232[0] 1781588169.624586 trunc from 0x8814fe to 0x8e7773 (0x66276 bytes) 000234[0] 1781588169.678692 trunc from 0x8e7774 to 0x8ac2ab (0x3b4c8 bytes) 000236[1] 1781588169.744736 mapread 0x6e7000 thru 0x6eed7b (0x7d7c bytes) 000237[1] 1781588169.789747 mapwrite 0x81a000 thru 0x827913 (0xd914 bytes) 000238[0] 1781588169.896732 trunc from 0x8ac2ac to 0x603089 (0x2a9222 bytes) 000240[1] 1781588169.972752 mapwrite 0x160000 thru 0x160ea2 (0xea3 bytes) 000244[1] 1781588170.111463 read_OD 0x597000 thru 0x59ffff (0x9000 bytes) 000246[0] 1781588170.162679 trunc from 0x60308a to 0x21eb8b (0x3e44fe bytes) 000252[0] 1781588170.269157 write_OD 0x004000 thru 0x012fff (0xf000 bytes) 000253[0] 1781588170.325128 mapread 0x185000 thru 0x18c9c6 (0x79c7 bytes) 000256[0] 1781588170.363411 trunc from 0x21eb8c to 0x26fe37 (0x512ac bytes) 000257[0] 1781588170.404184 mapwrite 0x932000 thru 0x9418f2 (0xf8f3 bytes) 000258[1] 1781588170.537204 write_OD 0x583000 thru 0x583fff (0x1000 bytes) 000259[1] 1781588170.589227 mapread 0x5bd000 thru 0x5cafd8 (0xdfd9 bytes) 000260[1] 1781588170.668708 mapread 0x532000 thru 0x538216 (0x6217 bytes) 000263[0] 1781588170.688177 mapwrite 0x8fe000 thru 0x90bd69 (0xdd6a bytes) 000265[0] 1781588170.783843 mapwrite 0x95c000 thru 0x968bd4 (0xcbd5 bytes) 000271[1] 1781588170.953067 mapread 0x5a2000 thru 0x5a77ff (0x5800 bytes) 000272[1] 1781588170.978274 trunc from 0x968bd5 to 0x5f0815 (0x3783bf bytes) 000273[0] 1781588171.007679 write_OD 0x5b4000 thru 0x5b8fff (0x5000 bytes) 000274[0] 1781588171.060147 mapread 0x4d4000 thru 0x4db647 (0x7648 bytes) 000277[1] 1781588171.086694 mapwrite 0x91c000 thru 0x922dd9 (0x6dda bytes) 000278[0] 1781588171.213304 mapwrite 0x53a000 thru 0x5490af (0xf0b0 bytes) 000279[1] 1781588171.326439 mapread 0x3b4000 thru 0x3b62a1 (0x22a2 bytes) 000280[0] 1781588171.391798 write_OD 0x612000 thru 0x615fff (0x4000 bytes) 000283[0] 1781588171.458325 write_OD 0x514000 thru 0x519fff (0x6000 bytes) 000284[0] 1781588171.526328 read_OD 0x4a4000 thru 0x4affff (0xc000 bytes) 000285[1] 1781588171.542379 mapwrite 0x7cd000 thru 0x7cf16d (0x216e bytes) 000287[1] 1781588171.642179 trunc from 0x922dda to 0x662f2b (0x2bfeae bytes) 000289[0] 1781588171.664085 trunc from 0x662f2c to 0x6102a3 (0x52c88 bytes) 000291[1] 1781588171.721790 mapwrite 0x697000 thru 0x69fec9 (0x8eca bytes) 000292[0] 1781588171.871550 write_OD 0x0a0000 thru 0x0a5fff (0x6000 bytes) 000294[0] 1781588171.957714 read_OD 0x1e3000 thru 0x1eafff (0x8000 bytes) 000295[0] 1781588171.967742 read_OD 0x400000 thru 0x40dfff (0xe000 bytes) 000297[0] 1781588171.983252 write_OD 0x60a000 thru 0x610fff (0x7000 bytes) 000300[0] 1781588172.056405 trunc from 0x69feca to 0x29a001 (0x405ec8 bytes) 000301[1] 1781588172.134430 read_OD 0x0d0000 thru 0x0d3fff (0x4000 bytes) 000304[0] 1781588172.177808 trunc from 0x29a002 to 0x6a3d6d (0x409d6c bytes) 000305[0] 1781588172.210667 mapwrite 0x29d000 thru 0x2a9371 (0xc372 bytes) 000306[1] 1781588172.317176 mapwrite 0x8bb000 thru 0x8bc809 (0x180a bytes) 000308[1] 1781588172.560672 read_OD 0x354000 thru 0x35efff (0xb000 bytes) 000311[0] 1781588172.608637 trunc from 0x8bc80a to 0x6887b3 (0x234056 bytes) 000313[1] 1781588172.668209 trunc from 0x6887b4 to 0x2e1924 (0x3a6e8f bytes) 000314[0] 1781588172.702014 mapwrite 0x29f000 thru 0x2a37fa (0x47fb bytes) 000315[1] 1781588172.841691 write_OD 0x1d8000 thru 0x1dbfff (0x4000 bytes) 000318[0] 1781588172.912781 read_OD 0x22b000 thru 0x237fff (0xd000 bytes) 000319[1] 1781588172.921163 mapread 0x12d000 thru 0x12e31b (0x131c bytes) 000323[0] 1781588172.942302 mapread 0x11e000 thru 0x1245a2 (0x65a3 bytes) 000327[1] 1781588173.005692 mapread 0x15e000 thru 0x16ac9e (0xcc9f bytes) truncating to largest ever: 0x9da866 000328[0] 1781588173.031498 trunc from 0x2e1925 to 0x9da865 (0x6f8f41 bytes) 000329[1] 1781588173.100707 read_OD 0x8a4000 thru 0x8affff (0xc000 bytes) 000331[0] 1781588173.138565 trunc from 0x9da866 to 0x991408 (0x4945d bytes) 000332[1] 1781588173.177056 read_OD 0x650000 thru 0x650fff (0x1000 bytes) 000333[1] 1781588173.200493 write_OD 0x92d000 thru 0x938fff (0xc000 bytes) 000335[1] 1781588173.255679 read_OD 0x242000 thru 0x24afff (0x9000 bytes) 000337[1] 1781588173.268237 mapwrite 0x8e0000 thru 0x8e91c2 (0x91c3 bytes) 000338[0] 1781588173.401818 read_OD 0x20a000 thru 0x20ffff (0x6000 bytes) 000339[1] 1781588173.435643 read_OD 0x641000 thru 0x642fff (0x2000 bytes) 000340[0] 1781588173.445036 mapwrite 0x3f5000 thru 0x3f7e04 (0x2e05 bytes) 000341[0] 1781588173.514154 mapread 0x80d000 thru 0x8193a4 (0xc3a5 bytes) 000342[1] 1781588173.572657 read_OD 0x74f000 thru 0x756fff (0x8000 bytes) 000344[0] 1781588173.619984 write_OD 0x191000 thru 0x19cfff (0xc000 bytes) 000345[1] 1781588173.682817 read_OD 0x3b4000 thru 0x3b8fff (0x5000 bytes) 000346[0] 1781588173.695390 write_OD 0x255000 thru 0x25ffff (0xb000 bytes) 000347[1] 1781588173.769415 read_OD 0x0ca000 thru 0x0d7fff (0xe000 bytes) 000351[0] 1781588173.791724 mapwrite 0x4d3000 thru 0x4d8584 (0x5585 bytes) 000352[1] 1781588173.984257 mapread 0x182000 thru 0x18c9ef (0xa9f0 bytes) 000353[0] 1781588174.065216 write_OD 0x95f000 thru 0x963fff (0x5000 bytes) 000356[0] 1781588174.152917 write_OD 0x68f000 thru 0x68ffff (0x1000 bytes) 000362[0] 1781588174.231362 read_OD 0x092000 thru 0x098fff (0x7000 bytes) 000363[0] 1781588174.240966 mapwrite 0x2f1000 thru 0x2f8ffa (0x7ffb bytes) 000366[1] 1781588174.408454 write_OD 0x6d6000 thru 0x6e4fff (0xf000 bytes) 000367[1] 1781588174.471844 read_OD 0x481000 thru 0x48efff (0xe000 bytes) 000373[0] 1781588174.493428 write_OD 0x993000 thru 0x99cfff (0xa000 bytes) 000375[1] 1781588174.559533 trunc from 0x99d000 to 0x825c76 (0x177389 bytes) 000376[0] 1781588174.631115 trunc from 0x825c77 to 0x7e2ddc (0x42e9a bytes) 000377[0] 1781588174.668717 write_OD 0x88a000 thru 0x896fff (0xd000 bytes) 000383[1] 1781588174.775274 mapwrite 0x228000 thru 0x23261b (0xa61c bytes) 000384[1] 1781588174.879181 mapwrite 0x52d000 thru 0x530c42 (0x3c43 bytes) 000387[1] 1781588174.996497 mapread 0x6e5000 thru 0x6f2075 (0xd076 bytes) 000388[0] 1781588175.063552 mapread 0x32b000 thru 0x339fef (0xeff0 bytes) 000391[0] 1781588175.114957 mapread 0x6aa000 thru 0x6b6fd7 (0xcfd8 bytes) 000392[1] 1781588175.157805 read_OD 0x5fc000 thru 0x5fffff (0x4000 bytes) 000395[0] 1781588175.176715 write_OD 0x116000 thru 0x11ffff (0xa000 bytes) 000396[1] 1781588175.249092 mapread 0x838000 thru 0x83f189 (0x718a bytes) 000397[1] 1781588175.272034 mapread 0x6a5000 thru 0x6af981 (0xa982 bytes) 000398[0] 1781588175.295533 read_OD 0x01f000 thru 0x026fff (0x8000 bytes) 000399[0] 1781588175.303048 mapread 0x857000 thru 0x860d36 (0x9d37 bytes) 000400[1] 1781588175.327139 mapread 0x4b7000 thru 0x4c519c (0xe19d bytes) 000402[1] 1781588175.376522 mapwrite 0x700000 thru 0x704be1 (0x4be2 bytes) 000403[0] 1781588175.496342 mapwrite 0x184000 thru 0x18c0f9 (0x80fa bytes) 000407[0] 1781588175.640635 trunc from 0x897000 to 0x4ea107 (0x3acef8 bytes) 000409[0] 1781588175.677393 mapwrite 0x152000 thru 0x1589ae (0x69af bytes) 000410[0] 1781588175.743630 mapwrite 0x565000 thru 0x56ae68 (0x5e69 bytes) 000412[1] 1781588175.931694 read_OD 0x1ff000 thru 0x204fff (0x6000 bytes) 000413[0] 1781588175.957771 trunc from 0x56ae69 to 0x92fc0a (0x3c4da2 bytes) 000417[0] 1781588176.055411 write_OD 0x623000 thru 0x624fff (0x2000 bytes) 000418[0] 1781588176.107761 mapread 0x6be000 thru 0x6c820c (0xa20d bytes) 000419[0] 1781588176.137459 write_OD 0x0a3000 thru 0x0a5fff (0x3000 bytes) 000421[0] 1781588176.205484 write_OD 0x178000 thru 0x17efff (0x7000 bytes) 000422[1] 1781588176.275055 write_OD 0x61e000 thru 0x627fff (0xa000 bytes) 000424[1] 1781588176.332376 read_OD 0x5bf000 thru 0x5c5fff (0x7000 bytes) 000425[0] 1781588176.345736 read_OD 0x0ef000 thru 0x0f7fff (0x9000 bytes) 000427[0] 1781588176.364963 mapread 0x33c000 thru 0x347537 (0xb538 bytes) 000428[1] 1781588176.421747 trunc from 0x92fc0b to 0x234594 (0x6fb676 bytes) 000429[0] 1781588176.466452 mapwrite 0x86e000 thru 0x86f0a1 (0x10a2 bytes) 000430[1] 1781588176.589025 trunc from 0x86f0a2 to 0x5f8727 (0x27697a bytes) 000431[0] 1781588176.614275 mapread 0x1d5000 thru 0x1d6d4a (0x1d4b bytes) 000432[1] 1781588176.645314 mapread 0x372000 thru 0x373da2 (0x1da3 bytes) 000436[0] 1781588176.671129 trunc from 0x5f8728 to 0x2da8f6 (0x31de31 bytes) 000440[0] 1781588176.762365 mapread 0x0a6000 thru 0x0b1d2c (0xbd2d bytes) 000441[1] 1781588176.806774 trunc from 0x2da8f7 to 0x5f8bcd (0x31e2d7 bytes) 000442[1] 1781588176.863895 read_OD 0x5da000 thru 0x5e0fff (0x7000 bytes) 000446[0] 1781588176.897684 read_OD 0x077000 thru 0x07dfff (0x7000 bytes) 000447[0] 1781588176.906016 mapread 0x2ba000 thru 0x2c8cea (0xeceb bytes) 000449[0] 1781588176.932468 mapwrite 0x955000 thru 0x95c5db (0x75dc bytes) 000451[0] 1781588177.041890 trunc from 0x95c5dc to 0x77e3ca (0x1de211 bytes) 000454[1] 1781588177.071464 write_OD 0x3cf000 thru 0x3dafff (0xc000 bytes) 000457[1] 1781588177.137043 mapread 0x57a000 thru 0x5803ac (0x63ad bytes) 000460[1] 1781588177.160469 mapwrite 0x57f000 thru 0x587d1e (0x8d1f bytes) 000462[0] 1781588177.269918 mapwrite 0x7e2000 thru 0x7ee2cf (0xc2d0 bytes) 000466[1] 1781588177.471588 mapwrite 0x9d9000 thru 0x9e8d69 (0xfd6a bytes) 000467[1] 1781588177.658392 read_OD 0x544000 thru 0x549fff (0x6000 bytes) 000471[1] 1781588177.706225 mapwrite 0x4df000 thru 0x4e3b93 (0x4b94 bytes) 000474[1] 1781588177.892273 mapread 0x880000 thru 0x886b5b (0x6b5c bytes) 000475[0] 1781588177.919073 trunc from 0x9e8d6a to 0x38217e (0x666beb bytes) 000476[1] 1781588177.982367 trunc from 0x38217f to 0x725860 (0x3a36e2 bytes) 000477[1] 1781588178.071270 mapwrite 0x342000 thru 0x346af4 (0x4af5 bytes) 000481[1] 1781588178.171753 trunc from 0x725861 to 0x576e7e (0x1ae9e2 bytes) 000482[1] 1781588178.218220 mapread 0x138000 thru 0x147397 (0xf398 bytes) 000484[0] 1781588178.266980 read_OD 0x225000 thru 0x229fff (0x5000 bytes) 000494[1] 1781588178.314043 read_OD 0x08c000 thru 0x091fff (0x6000 bytes) 000495[0] 1781588178.324510 mapread 0x365000 thru 0x367772 (0x2773 bytes) 000497[1] 1781588178.359937 write_OD 0x4ff000 thru 0x502fff (0x4000 bytes) 000498[1] 1781588178.429614 mapread 0x4b7000 thru 0x4c63a1 (0xf3a2 bytes) 000500[0] 1781588178.477244 write_OD 0x619000 thru 0x626fff (0xe000 bytes) All operations completed A-OK! PASS 16a (49s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 01:36:24 (1781588184) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.359515 s, 23.3 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.296669 s, 28.3 MB/s Chance of close/open is 1 in 50 Seed set to 8186 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1781588187.499637 mapwrite 0x747497 thru 0x748a4e (0x15b8 bytes) 000003[0] 1781588187.636766 write 0x954be5 thru 0x955058 (0x474 bytes) 000004[1] 1781588187.697211 read 0x584b9c thru 0x586412 (0x1877 bytes) 000007[1] 1781588187.780186 mapread 0x64e58a thru 0x65013e (0x1bb5 bytes) 000008[1] 1781588187.819834 write 0x8cd0e2 thru 0x8ce6d4 (0x15f3 bytes) 000009[1] 1781588187.882917 read 0x83e3bf thru 0x83e708 (0x34a bytes) truncating to largest ever: 0x3ae5b2 000010[0] 1781588187.939745 trunc from 0x955059 to 0x3ae5b1 (0x5a6aa7 bytes) 000011[0] 1781588188.008883 mapwrite 0x721060 thru 0x72170a (0x6ab bytes) 000014[0] 1781588188.183395 read 0x6fe5fd thru 0x6feb3b (0x53f bytes) 000017[0] 1781588188.214829 mapwrite 0x075e05 thru 0x076d37 (0xf33 bytes) 000018[1] 1781588188.327751 read 0x2b0b02 thru 0x2b2650 (0x1b4f bytes) 000021[0] 1781588188.401463 mapread 0x706411 thru 0x707b81 (0x1771 bytes) 000022[1] 1781588188.418300 read 0x0c6ce1 thru 0x0c8752 (0x1a72 bytes) 000024[0] 1781588188.451411 mapwrite 0x21c37b thru 0x21dac6 (0x174c bytes) 000025[1] 1781588188.543826 write 0x3dda9b thru 0x3defdc (0x1542 bytes) 000027[1] 1781588188.620281 write 0x226097 thru 0x226957 (0x8c1 bytes) 000028[1] 1781588188.653294 write 0x23aa08 thru 0x23c2ed (0x18e6 bytes) 000030[0] 1781588188.711247 write 0x5354ac thru 0x53566d (0x1c2 bytes) 000033[1] 1781588188.769838 mapread 0x6d9c1c thru 0x6da83d (0xc22 bytes) 000035[0] 1781588188.828539 write 0x049992 thru 0x04b718 (0x1d87 bytes) 000036[0] 1781588188.883627 read 0x26c03f thru 0x26c80b (0x7cd bytes) 000040[1] 1781588188.894025 mapread 0x223874 thru 0x224083 (0x810 bytes) 000041[0] 1781588188.937791 write 0x40b034 thru 0x40bd1c (0xce9 bytes) 000042[0] 1781588188.960664 mapwrite 0x2a44a2 thru 0x2a62d1 (0x1e30 bytes) 000044[0] 1781588189.110914 read 0x311a65 thru 0x312f4e (0x14ea bytes) 000045[1] 1781588189.123440 read 0x16c48f thru 0x16d860 (0x13d2 bytes) truncating to largest ever: 0x610931 000047[1] 1781588189.154980 trunc from 0x72170b to 0x610930 (0x110dda bytes) 000050[1] 1781588189.315968 read 0x4c3167 thru 0x4c4866 (0x1700 bytes) 000051[0] 1781588189.365049 read 0x044e72 thru 0x046e05 (0x1f94 bytes) 000052[1] 1781588189.404887 write 0x354946 thru 0x3556e8 (0xda3 bytes) 000055[1] 1781588189.482371 write 0x7312e9 thru 0x731bee (0x906 bytes) 000057[0] 1781588189.525674 write 0x3900bc thru 0x391d34 (0x1c79 bytes) 000058[1] 1781588189.571638 read 0x47ed0d thru 0x47f24d (0x541 bytes) 000059[0] 1781588189.589312 mapread 0x29c39b thru 0x29cfd8 (0xc3e bytes) 000061[0] 1781588189.654950 trunc from 0x731bef to 0x2a6520 (0x48b6ce bytes) 000062[0] 1781588189.699187 write 0x3091e2 thru 0x309b08 (0x927 bytes) 000064[0] 1781588189.740629 trunc from 0x309b09 to 0x29a24d (0x6f8bb bytes) 000067[0] 1781588189.876044 mapwrite 0x627218 thru 0x628a5a (0x1843 bytes) 000068[0] 1781588190.069209 mapread 0x228976 thru 0x22a650 (0x1cdb bytes) truncating to largest ever: 0x90e887 000069[0] 1781588190.140967 trunc from 0x628a5b to 0x90e886 (0x2e5e2c bytes) 000071[1] 1781588190.229352 write 0x4509f8 thru 0x4526f5 (0x1cfe bytes) 000072[0] 1781588190.291348 mapread 0x0e3d23 thru 0x0e45da (0x8b8 bytes) 000076[0] 1781588190.452071 write 0x8f1859 thru 0x8f238d (0xb35 bytes) 000081[1] 1781588190.551617 read 0x6860e2 thru 0x686abb (0x9da bytes) 000083[1] 1781588190.599912 trunc from 0x90e887 to 0x56073f (0x3ae147 bytes) 000084[1] 1781588190.658344 mapread 0x354aaf thru 0x35656c (0x1abe bytes) 000085[1] 1781588190.730047 mapwrite 0x321e77 thru 0x321f74 (0xfe bytes) 000086[0] 1781588190.786685 mapwrite 0x0cd27b thru 0x0ce55c (0x12e2 bytes) 000087[1] 1781588190.888995 mapread 0x1eaa18 thru 0x1ec870 (0x1e59 bytes) 000089[1] 1781588190.937347 write 0x9418dd thru 0x9427ea (0xf0e bytes) 000090[1] 1781588190.948393 trunc from 0x9427eb to 0x1fcf18 (0x7458d2 bytes) 000091[0] 1781588190.999499 mapread 0x144bc6 thru 0x1462f8 (0x1733 bytes) 000093[1] 1781588191.066975 write 0x25b39d thru 0x25ba10 (0x674 bytes) 000095[1] 1781588191.130523 read 0x1facda thru 0x1fb726 (0xa4d bytes) 000096[1] 1781588191.212806 read 0x215945 thru 0x216604 (0xcc0 bytes) 000101[1] 1781588191.244127 read 0x249f2e thru 0x24ab73 (0xc46 bytes) 000102[0] 1781588191.245725 read 0x0496a2 thru 0x049ffc (0x95b bytes) 000103[0] 1781588191.343097 read 0x25692b thru 0x257836 (0xf0c bytes) 000104[0] 1781588191.351324 write 0x490e05 thru 0x49211c (0x1318 bytes) 000105[0] 1781588191.385690 read 0x0e980b thru 0x0ea443 (0xc39 bytes) 000106[1] 1781588191.390021 mapwrite 0x83c9f3 thru 0x83ca64 (0x72 bytes) 000107[0] 1781588191.568152 trunc from 0x83ca65 to 0x59d948 (0x29f11c bytes) 000108[0] 1781588191.621518 trunc from 0x59d949 to 0x56cf8e (0x309ba bytes) 000110[1] 1781588191.640439 mapwrite 0x42da7c thru 0x42f7aa (0x1d2f bytes) 000111[0] 1781588191.762609 write 0x3ac886 thru 0x3ae6d6 (0x1e51 bytes) 000114[1] 1781588191.893597 mapread 0x0e0340 thru 0x0e0870 (0x531 bytes) 000115[1] 1781588191.941603 mapwrite 0x71ae80 thru 0x71b2a3 (0x424 bytes) 000117[0] 1781588192.088994 trunc from 0x71b2a4 to 0x8acb10 (0x19186d bytes) 000118[0] 1781588192.179880 trunc from 0x8acb11 to 0x045cf0 (0x866e20 bytes) 000119[0] 1781588192.222467 mapwrite 0x1aba9e thru 0x1ac3ae (0x911 bytes) 000120[1] 1781588192.379090 trunc from 0x1ac3af to 0x4d445c (0x3280ae bytes) 000121[0] 1781588192.415379 mapread 0x2ec48e thru 0x2ed654 (0x11c7 bytes) 000124[0] 1781588192.450663 trunc from 0x4d445d to 0x6f8edd (0x224a81 bytes) 000128[0] 1781588192.562968 mapread 0x559bd6 thru 0x559bf9 (0x24 bytes) 000129[1] 1781588192.603788 read 0x43631d thru 0x436de3 (0xac7 bytes) 000130[1] 1781588192.666657 mapread 0x3087c9 thru 0x30906c (0x8a4 bytes) 000131[1] 1781588192.713948 mapread 0x571283 thru 0x5730a5 (0x1e23 bytes) 000132[1] 1781588192.734026 mapwrite 0x10109c thru 0x101e8b (0xdf0 bytes) 000133[1] 1781588192.838180 write 0x9e9855 thru 0x9ea8f6 (0x10a2 bytes) 000136[0] 1781588192.889170 read 0x5f7025 thru 0x5f7e3a (0xe16 bytes) 000137[0] 1781588192.895826 mapread 0x3b3e23 thru 0x3b438e (0x56c bytes) 000142[0] 1781588192.942615 write 0x7d805c thru 0x7d9155 (0x10fa bytes) 000145[0] 1781588192.989025 read 0x891295 thru 0x89210a (0xe76 bytes) 000147[0] 1781588193.030903 mapwrite 0x43e2c4 thru 0x43fa23 (0x1760 bytes) 000151[1] 1781588193.113056 mapwrite 0x7600a6 thru 0x761d2b (0x1c86 bytes) 000153[0] 1781588193.221787 mapread 0x826c48 thru 0x8276e3 (0xa9c bytes) 000154[0] 1781588193.244504 mapread 0x714e4e thru 0x7154bb (0x66e bytes) 000155[0] 1781588193.298743 trunc from 0x9ea8f7 to 0x82156c (0x1c938a bytes) 000156[1] 1781588193.411540 mapwrite 0x5acd3a thru 0x5ad3b0 (0x677 bytes) 000158[1] 1781588193.624983 write 0x2fc1f4 thru 0x2fc538 (0x345 bytes) 000160[1] 1781588193.714076 mapread 0x56d5e8 thru 0x56d667 (0x80 bytes) 000163[1] 1781588193.729093 mapread 0x4b6acd thru 0x4b6e41 (0x375 bytes) 000164[0] 1781588193.782390 read 0x654bbe thru 0x656bb5 (0x1ff8 bytes) 000165[0] 1781588193.852143 mapwrite 0x3df74a thru 0x3df7c7 (0x7e bytes) 000170[1] 1781588194.048331 read 0x632cb6 thru 0x633995 (0xce0 bytes) 000171[0] 1781588194.054258 mapwrite 0x0f75be thru 0x0f9149 (0x1b8c bytes) 000174[0] 1781588194.130982 write 0x28da60 thru 0x28e6d4 (0xc75 bytes) 000175[0] 1781588194.180058 trunc from 0x82156d to 0x61ef2c (0x202640 bytes) 000176[0] 1781588194.278376 write 0x4235ff thru 0x4239a9 (0x3ab bytes) 000177[0] 1781588194.322666 read 0x572d00 thru 0x573ef1 (0x11f2 bytes) 000180[0] 1781588194.361093 trunc from 0x61ef2d to 0x64e074 (0x2f148 bytes) 000184[0] 1781588194.443001 write 0x55f02e thru 0x560f67 (0x1f3a bytes) 000190[0] 1781588194.506785 mapwrite 0x50e9ba thru 0x5103c4 (0x1a0b bytes) 000191[1] 1781588194.610190 write 0x58c9bc thru 0x58db92 (0x11d7 bytes) 000193[0] 1781588194.698018 trunc from 0x64e075 to 0x7c144f (0x1733db bytes) 000196[0] 1781588194.766429 mapwrite 0x65a151 thru 0x65b43a (0x12ea bytes) 000197[0] 1781588194.854270 write 0x6c1bde thru 0x6c1fcb (0x3ee bytes) 000198[1] 1781588194.892762 write 0x4bd3bf thru 0x4bea54 (0x1696 bytes) truncating to largest ever: 0x960de3 000200[1] 1781588194.986056 trunc from 0x7c1450 to 0x960de2 (0x19f993 bytes) 000205[1] 1781588195.073635 trunc from 0x960de3 to 0x4ca08a (0x496d58 bytes) 000207[1] 1781588195.133834 mapread 0x007c6b thru 0x009a32 (0x1dc8 bytes) 000212[0] 1781588195.252935 mapwrite 0x5f254a thru 0x5f4411 (0x1ec8 bytes) 000213[0] 1781588195.413037 write 0x7e75bf thru 0x7e8e71 (0x18b3 bytes) 000219[1] 1781588195.492975 mapwrite 0x533707 thru 0x534266 (0xb60 bytes) 000221[1] 1781588195.705365 mapread 0x51f02c thru 0x520190 (0x1165 bytes) 000222[1] 1781588195.775677 trunc from 0x7e8e72 to 0x8863ca (0x9d559 bytes) 000223[0] 1781588195.824688 trunc from 0x8863cb to 0x87fa59 (0x6971 bytes) 000228[1] 1781588196.021037 mapwrite 0x0f99c4 thru 0x0fb387 (0x19c4 bytes) 000232[1] 1781588196.299595 mapread 0x0b6cea thru 0x0b771c (0xa33 bytes) 000233[1] 1781588196.360374 mapread 0x7aa64e thru 0x7ac5de (0x1f91 bytes) 000236[1] 1781588196.423464 write 0x0ad8bc thru 0x0af368 (0x1aad bytes) 000238[1] 1781588196.442787 mapread 0x5dd5fe thru 0x5df075 (0x1a78 bytes) 000239[0] 1781588196.473081 write 0x281fab thru 0x283764 (0x17ba bytes) 000240[1] 1781588196.511767 trunc from 0x87fa5a to 0x0bdac8 (0x7c1f91 bytes) 000242[0] 1781588196.587956 mapwrite 0x5cc71c thru 0x5cd0d5 (0x9ba bytes) 000243[1] 1781588196.702023 trunc from 0x5cd0d6 to 0x68b617 (0xbe542 bytes) 000244[1] 1781588196.769263 trunc from 0x68b618 to 0x07fc01 (0x60ba16 bytes) 000246[0] 1781588196.851777 trunc from 0x07fc02 to 0x7a11e4 (0x7215e3 bytes) 000250[1] 1781588197.058543 write 0x89b2ed thru 0x89b5a5 (0x2b9 bytes) 000251[0] 1781588197.101021 write 0x01d0f6 thru 0x01dd44 (0xc4f bytes) 000253[0] 1781588197.192137 write 0x1c2161 thru 0x1c309b (0xf3b bytes) 000255[1] 1781588197.292159 read 0x0f2f75 thru 0x0f3ff5 (0x1081 bytes) 000258[1] 1781588197.373822 write 0x183840 thru 0x184e93 (0x1654 bytes) 000260[1] 1781588197.514023 mapread 0x24022a thru 0x24062d (0x404 bytes) 000261[1] 1781588197.556323 read 0x42237a thru 0x422986 (0x60d bytes) 000265[0] 1781588197.603372 trunc from 0x89b5a6 to 0x4d28e3 (0x3c8cc2 bytes) 000266[0] 1781588197.664787 trunc from 0x4d28e4 to 0x4d19e4 (0xeff bytes) 000271[1] 1781588197.840754 read 0x1a09a2 thru 0x1a15f4 (0xc53 bytes) 000272[0] 1781588197.898389 read 0x2f3fcd thru 0x2f5406 (0x143a bytes) 000274[1] 1781588197.946104 mapwrite 0x970de3 thru 0x971f2e (0x114c bytes) 000275[0] 1781588198.177172 mapread 0x87a720 thru 0x87aa7b (0x35c bytes) 000277[0] 1781588198.236821 read 0x28ad66 thru 0x28add3 (0x6e bytes) 000278[1] 1781588198.256336 mapwrite 0x8cc982 thru 0x8ce962 (0x1fe1 bytes) 000279[0] 1781588198.371395 trunc from 0x971f2f to 0x041ef0 (0x93003e bytes) 000282[0] 1781588198.554550 read 0x03f546 thru 0x0402b9 (0xd74 bytes) 000283[1] 1781588198.593758 write 0x106ffa thru 0x108469 (0x1470 bytes) 000284[0] 1781588198.636964 write 0x52ec2d thru 0x530616 (0x19ea bytes) 000285[0] 1781588198.666597 mapwrite 0x776de2 thru 0x77708b (0x2aa bytes) 000288[1] 1781588199.127831 mapread 0x637c7e thru 0x637f8c (0x30f bytes) 000290[0] 1781588199.238339 mapread 0x20532d thru 0x20695e (0x1632 bytes) 000292[0] 1781588199.241535 write 0x99f934 thru 0x9a04f8 (0xbc5 bytes) 000294[1] 1781588199.271812 write 0x7e6ee1 thru 0x7e8a6b (0x1b8b bytes) 000296[1] 1781588199.382362 read 0x60d809 thru 0x60de22 (0x61a bytes) 000297[0] 1781588199.433784 read 0x051dc9 thru 0x0526e7 (0x91f bytes) 000298[0] 1781588199.467705 trunc from 0x9a04f9 to 0x2e021f (0x6c02d9 bytes) 000299[1] 1781588199.514095 read 0x0d5f56 thru 0x0d69eb (0xa96 bytes) 000301[0] 1781588199.557528 mapread 0x040233 thru 0x041b02 (0x18d0 bytes) 000302[1] 1781588199.594946 trunc from 0x2e0220 to 0x88b580 (0x5ab361 bytes) 000310[0] 1781588199.742415 read 0x0e427e thru 0x0e51be (0xf41 bytes) 000311[1] 1781588199.813989 trunc from 0x88b581 to 0x5459f1 (0x345b8f bytes) 000313[1] 1781588199.890482 mapwrite 0x3a80fb thru 0x3a88b5 (0x7bb bytes) 000315[0] 1781588199.982828 read 0x2607a9 thru 0x261814 (0x106c bytes) 000318[1] 1781588200.038069 read 0x061284 thru 0x062967 (0x16e4 bytes) 000319[1] 1781588200.090371 mapread 0x2a4017 thru 0x2a4dde (0xdc8 bytes) 000320[1] 1781588200.125735 write 0x26e96c thru 0x26f737 (0xdcc bytes) 000323[0] 1781588200.258047 write 0x41aa0a thru 0x41bae4 (0x10db bytes) 000326[1] 1781588200.376289 read 0x17d36a thru 0x17e945 (0x15dc bytes) 000329[0] 1781588200.470319 write 0x383509 thru 0x3845e4 (0x10dc bytes) 000331[1] 1781588200.523973 trunc from 0x5459f2 to 0x7c6d9d (0x2813ac bytes) 000333[1] 1781588200.698724 write 0x34e68a thru 0x34fb33 (0x14aa bytes) 000334[1] 1781588200.725509 mapread 0x76a997 thru 0x76c1cb (0x1835 bytes) 000335[1] 1781588200.803628 write 0x447e56 thru 0x447e73 (0x1e bytes) 000338[1] 1781588200.876641 read 0x32ef2b thru 0x33091c (0x19f2 bytes) 000340[1] 1781588200.969071 read 0x0ef014 thru 0x0efa0a (0x9f7 bytes) 000344[0] 1781588200.979842 read 0x4e6209 thru 0x4e6dfe (0xbf6 bytes) 000345[0] 1781588201.011595 mapwrite 0x97a135 thru 0x97b674 (0x1540 bytes) 000347[0] 1781588201.173317 trunc from 0x97b675 to 0x08547f (0x8f61f5 bytes) 000349[1] 1781588201.237225 mapread 0x01b1ae thru 0x01b499 (0x2ec bytes) 000350[0] 1781588201.265289 mapread 0x07f5ed thru 0x07fd68 (0x77c bytes) 000353[1] 1781588201.313741 write 0x7aad2d thru 0x7ac93e (0x1c12 bytes) 000355[1] 1781588201.349765 mapwrite 0x993939 thru 0x9947df (0xea7 bytes) truncating to largest ever: 0x9a770f 000356[1] 1781588201.455821 trunc from 0x9947e0 to 0x9a770e (0x12f2f bytes) 000358[1] 1781588201.479546 mapwrite 0x212830 thru 0x2129ac (0x17d bytes) 000359[1] 1781588201.588038 trunc from 0x9a770f to 0x052989 (0x954d85 bytes) 000361[1] 1781588201.672003 mapread 0x027502 thru 0x028783 (0x1282 bytes) 000362[0] 1781588201.736542 write 0x2cc741 thru 0x2cccf4 (0x5b4 bytes) 000363[0] 1781588201.789125 write 0x146c11 thru 0x1488f4 (0x1ce4 bytes) 000364[0] 1781588201.822984 mapwrite 0x23a8ac thru 0x23c6c4 (0x1e19 bytes) 000365[0] 1781588201.948173 mapwrite 0x54ac28 thru 0x54babf (0xe98 bytes) 000367[1] 1781588202.154902 mapwrite 0x3fbc83 thru 0x3fc5b2 (0x930 bytes) 000368[1] 1781588202.326686 read 0x3d4725 thru 0x3d5527 (0xe03 bytes) 000369[0] 1781588202.360596 read 0x020ac8 thru 0x020e1f (0x358 bytes) 000370[0] 1781588202.406397 mapread 0x13f313 thru 0x141220 (0x1f0e bytes) 000371[1] 1781588202.441535 read 0x2fe610 thru 0x2ff8ee (0x12df bytes) 000375[0] 1781588202.538889 write 0x8a22fb thru 0x8a23fa (0x100 bytes) 000376[0] 1781588202.563971 read 0x17e462 thru 0x17f5bf (0x115e bytes) 000378[0] 1781588202.640825 trunc from 0x8a23fb to 0x20e38d (0x69406d bytes) 000381[1] 1781588202.712066 mapwrite 0x6f9502 thru 0x6fa849 (0x1348 bytes) 000383[0] 1781588202.858409 trunc from 0x6fa84a to 0x8e9899 (0x1ef050 bytes) 000387[1] 1781588202.985606 write 0x726b48 thru 0x727fec (0x14a5 bytes) 000388[1] 1781588203.065084 write 0x80f202 thru 0x8101f6 (0xff5 bytes) 000389[0] 1781588203.142941 write 0x1d5906 thru 0x1d719f (0x189a bytes) 000390[1] 1781588203.211991 trunc from 0x8e989a to 0x5f01dc (0x2f96bd bytes) 000393[1] 1781588203.351399 mapwrite 0x1d9c30 thru 0x1daaf2 (0xec3 bytes) 000396[1] 1781588203.490784 write 0x5bc2f9 thru 0x5bc7a1 (0x4a9 bytes) 000397[0] 1781588203.533887 write 0x27fe72 thru 0x281cfb (0x1e8a bytes) 000398[0] 1781588203.596713 trunc from 0x5f01dd to 0x4db515 (0x114cc7 bytes) 000400[0] 1781588203.633947 read 0x369ff4 thru 0x36a4a0 (0x4ad bytes) 000402[0] 1781588203.698084 mapread 0x11fb78 thru 0x12015f (0x5e8 bytes) 000403[1] 1781588203.711489 trunc from 0x4db516 to 0x07bbb6 (0x45f95f bytes) 000404[0] 1781588203.793085 read 0x064f74 thru 0x065c46 (0xcd3 bytes) 000406[1] 1781588203.852348 mapwrite 0x9fd9e8 thru 0x9fed25 (0x133e bytes) 000407[0] 1781588203.975421 read 0x1ba6f1 thru 0x1bb69a (0xfaa bytes) 000410[0] 1781588204.041745 mapread 0x21281d thru 0x21461a (0x1dfe bytes) 000411[1] 1781588204.077743 mapwrite 0x078e91 thru 0x079a11 (0xb81 bytes) 000414[1] 1781588204.239258 read 0x047b10 thru 0x048a58 (0xf49 bytes) 000416[0] 1781588204.315111 trunc from 0x9fed26 to 0x92c65d (0xd26c8 bytes) 000420[0] 1781588204.456292 mapread 0x2dd6e6 thru 0x2defde (0x18f9 bytes) 000421[1] 1781588204.473271 write 0x7d823c thru 0x7d883b (0x600 bytes) 000426[0] 1781588204.699422 write 0x5708c1 thru 0x572322 (0x1a62 bytes) 000427[1] 1781588204.727941 trunc from 0x92c65e to 0x02667f (0x905fde bytes) 000429[1] 1781588204.795705 read 0x006360 thru 0x007f27 (0x1bc8 bytes) 000430[1] 1781588204.812089 mapread 0x00b534 thru 0x00c8d4 (0x13a1 bytes) 000431[1] 1781588204.813083 mapread 0x021e1e thru 0x02338b (0x156e bytes) 000433[0] 1781588204.851542 mapwrite 0x3414ca thru 0x3432d0 (0x1e07 bytes) 000434[1] 1781588205.025735 mapwrite 0x8fe781 thru 0x8ff967 (0x11e7 bytes) 000435[0] 1781588205.237510 read 0x5388f5 thru 0x53a12e (0x183a bytes) 000436[1] 1781588205.343147 write 0x6eb759 thru 0x6eb98d (0x235 bytes) 000439[0] 1781588205.367294 mapwrite 0x61567e thru 0x616907 (0x128a bytes) 000440[1] 1781588205.533068 read 0x68db47 thru 0x68e905 (0xdbf bytes) 000441[0] 1781588205.559163 read 0x295836 thru 0x295b7c (0x347 bytes) 000443[1] 1781588205.609121 mapwrite 0x7520e5 thru 0x754038 (0x1f54 bytes) 000445[1] 1781588205.800320 mapwrite 0x824baa thru 0x8260eb (0x1542 bytes) 000449[1] 1781588205.934235 read 0x17597f thru 0x175c08 (0x28a bytes) 000453[0] 1781588205.973774 mapread 0x7b1c0a thru 0x7b1fde (0x3d5 bytes) 000454[1] 1781588206.010248 trunc from 0x8ff968 to 0x6763d1 (0x289596 bytes) 000455[1] 1781588206.050759 mapwrite 0x6ee2ab thru 0x6eef02 (0xc58 bytes) 000456[0] 1781588206.197158 mapwrite 0x154d41 thru 0x156775 (0x1a35 bytes) 000458[1] 1781588206.430829 mapwrite 0x08bee8 thru 0x08c4dc (0x5f5 bytes) 000459[1] 1781588206.531310 trunc from 0x6eef03 to 0x2282eb (0x4c6c17 bytes) 000460[1] 1781588206.574406 read 0x00fa10 thru 0x010093 (0x684 bytes) 000463[1] 1781588206.653330 write 0x74b86a thru 0x74bb70 (0x307 bytes) 000466[1] 1781588206.682403 trunc from 0x74bb71 to 0x90b2a9 (0x1bf739 bytes) 000467[0] 1781588206.742150 read 0x8ea476 thru 0x8eb0af (0xc3a bytes) 000469[0] 1781588206.779209 mapread 0x8c31ad thru 0x8c3e82 (0xcd6 bytes) 000471[0] 1781588206.796983 mapread 0x192101 thru 0x192309 (0x209 bytes) 000472[0] 1781588206.814207 trunc from 0x90b2aa to 0x2a8a0f (0x66289a bytes) 000475[0] 1781588206.926735 mapread 0x155fc2 thru 0x1576ab (0x16ea bytes) 000476[1] 1781588206.942360 write 0x87167a thru 0x872e8e (0x1815 bytes) 000479[1] 1781588207.016109 mapwrite 0x8702b4 thru 0x870a59 (0x7a6 bytes) 000482[1] 1781588207.114326 mapread 0x387228 thru 0x3889fd (0x17d6 bytes) 000483[0] 1781588207.178991 mapwrite 0x613323 thru 0x6142a5 (0xf83 bytes) 000484[1] 1781588207.313250 mapwrite 0x467a88 thru 0x4683a7 (0x920 bytes) 000486[1] 1781588207.402485 write 0x9e2386 thru 0x9e2be7 (0x862 bytes) 000487[1] 1781588207.433526 mapwrite 0x80fd47 thru 0x811022 (0x12dc bytes) 000489[1] 1781588207.573023 mapwrite 0x8ebf88 thru 0x8ec535 (0x5ae bytes) 000490[1] 1781588207.655293 trunc from 0x9e2be8 to 0x5f14c7 (0x3f1720 bytes) 000493[0] 1781588207.769472 write 0x930cc4 thru 0x93186e (0xbab bytes) 000495[1] 1781588207.873246 mapread 0x9010a3 thru 0x901c85 (0xbe3 bytes) 000500[1] 1781588208.011530 mapread 0x09fe85 thru 0x0a100b (0x1187 bytes) All operations completed A-OK! PASS 16b (29s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 01:36:53 (1781588213) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 01:36:57 (1781588217) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.379513 s, 22.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.345872 s, 24.3 MB/s ldlm.namespaces.MGC192.168.206.144@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff93d0c2feb000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff93d0c5589800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff93d0c5589800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff93d0c5589800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.381 s, 9.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.81526 s, 27.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.44135 s, 30.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.0639 s, 14.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2079 s, 10.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.18286 s, 25.1 MB/s PASS 16d (54s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 01:37:51 (1781588271) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.341245 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204763 s, 5.1 MB/s PASS 16e (8s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 01:37:59 (1781588279) TIMEOUT PASS 16f (29s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 01:38:28 (1781588308) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 01:38:56 (1781588336) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 01:39:03 (1781588343) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 01:39:11 (1781588351) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.28562 s, 11.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.60819 s, 16.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.75027 s, 9.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.74795 s, 15.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.49524 s, 10.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.84144 s, 14.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.22471 s, 11.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.40983 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.49989 s, 7.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.78744 s, 6.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.90475 s, 6.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.16647 s, 8.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.22533 s, 8.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.06193 s, 6.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.35003 s, 6.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.72798 s, 9.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.29882 s, 7.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.27434 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.42102 s, 7.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (42s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 01:39:53 (1781588393) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Jun 16 01:40:37 EDT 2026 + fsx 2 done: rc=0 Tue Jun 16 01:40:37 EDT 2026 All operations completed A-OK! + fsx 3 done: rc=0 Tue Jun 16 01:40:37 EDT 2026 + fsx 4 done: rc=0 Tue Jun 16 01:40:37 EDT 2026 + fsx 5 done: rc=0 Tue Jun 16 01:40:37 EDT 2026 PASS 16k (54s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 01:40:47 (1781588447) fail_loc=0x8000030a PASS 17 (12s) == sanityn test 18: mmap sanity check =========================================================================================== 01:40:59 (1781588459) mmap test1: basic mmap operation (PASS, 0.17262s) mmap test2: MAP_PRIVATE not write back (PASS, 0.14183s) mmap test3: concurrent mmap ops on two nodes (PASS, 19.398s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.867s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.27812s) mmap test6: check mmap write/read content on two nodes (PASS, 0.26185s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 2.4136s) mmap test11: multi-thread shared mmap (PASS, 2.1116s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1258s) PASS 18 (40s) == sanityn test 19: test concurrent uncached read races ========================================================================= 01:41:39 (1781588499) oleg644-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg644-client: oleg644-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (5s) == sanityn test 20: test extra readahead page left in cache ============================================================== 01:41:44 (1781588504) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 01:41:51 (1781588511) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 01:42:00 (1781588520) now is 1781588522 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 new atime is 1781588584 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 01:43:10 (1781588590) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 01:43:16 (1781588596) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 01:43:23 (1781588603) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 01:43:31 (1781588611) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 01:43:34 (1781588614) utime: good mknod times 1781588614 <= 1781588615 <= 1781588616 for /mnt/lustre/f26a utime: good mknod times 1781588614 <= 1781588615 <= 1781588616 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 01:43:42 (1781588622) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 01:43:51 (1781588631) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-16 01:43:53.000000000 -0400 Birth: 2026-06-16 01:43:54.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-16 01:43:53.000000000 -0400 Birth: 2026-06-16 01:43:54.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 01:44:02 (1781588642) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 01:44:11 (1781588651) fail_loc=0x314 PASS 31a (9s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 01:44:21 (1781588661) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg644-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.144@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 01:44:45 (1781588685) -rw-r--r-- 1 root root 0 Jun 16 01:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 01:44:55 (1781588695) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43102 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 01:45:02 (1781588702) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 01:45:10 (1781588710) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (4s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 01:45:14 (1781588714) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 01:45:17 (1781588717) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 01:45:21 (1781588721) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 01:45:24 (1781588724) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93d0c2feb000.ost_server_uuid in IDLE state after 0 sec PASS 34 (145s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 01:47:49 (1781588869) fail_loc=0x80000317 Race attempt 0 Wait for 49285 49346 for 60 sec... - open/close 650 (time 1781588886.01 total 10.00 last 64.97) - open/close 1301 (time 1781588896.01 total 20.01 last 65.06) - open/close 1911 (time 1781588906.02 total 30.01 last 60.98) - open/close 2557 (time 1781588916.02 total 40.02 last 64.55) - open/close 3256 (time 1781588926.04 total 50.03 last 69.82) total: 4000 open/close in 59.54 seconds: 67.18 ops/second /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1746: 49285 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.144@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.144@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (72s) == sanityn test 36: handle ESTALE/open-unlink correctly == 01:49:01 (1781588941) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg644-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.54733 s, 33.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.29689 s, 40.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.36427 s, 38.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43621 s, 36.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.82907 s, 28.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51376 s, 34.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45797 s, 36.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48421 s, 35.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.43387 s, 36.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.60132 s, 32.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.1085 s, 47.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7442432) after(7493632) PASS 36 (300s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 01:54:01 (1781589241) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 - create 1369 (time 1781589256.39 total 10.00 last 136.89) - create 2858 (time 1781589266.40 total 20.00 last 148.85) - create 4152 (time 1781589276.40 total 30.01 last 129.37) - create 5597 (time 1781589286.40 total 40.01 last 144.45) - create 6914 (time 1781589296.41 total 50.02 last 131.56) - create 8092 (time 1781589306.41 total 60.02 last 117.79) - create 9403 (time 1781589316.43 total 70.04 last 130.90) total: 10000 create in 74.44 seconds: 134.34 ops/second PASS 37 (85s) == sanityn test 39a: file mtime does not change after rename ========================================================== 01:55:26 (1781589326) repeat after cancel_lru_locks PASS 39a (6s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 01:55:32 (1781589332) repeat after cancel_lru_locks PASS 39b (8s) == sanityn test 39c: check truncate mtime update ================================================================================ 01:55:40 (1781589340) repeat after cancel_lru_locks PASS 39c (7s) == sanityn test 39d: sync write should update mtime ====== 01:55:47 (1781589347) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 01:55:55 (1781589355) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (17s) == sanityn test 40c: pdirops: link and others ======================================================================== 01:56:12 (1781589372) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 01:56:30 (1781589390) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (20s) == sanityn test 40e: pdirops: rename and others ======================================================================== 01:56:50 (1781589410) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (16s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 01:57:06 (1781589426) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 01:57:21 (1781589441) open: File exists Conflict PASS 41b (15s) == sanityn test 41c: pdirops: create vs link ======================================================================== 01:57:36 (1781589456) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 01:57:50 (1781589470) Conflict PASS 41d (15s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 01:58:05 (1781589485) Conflict PASS 41e (12s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 01:58:18 (1781589498) Conflict PASS 41f (13s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 01:58:31 (1781589511) Conflict PASS 41g (15s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 01:58:46 (1781589526) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 01:59:00 (1781589540) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1051s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 02:16:31 (1781590591) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (13s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 02:16:44 (1781590604) open: File exists Conflict PASS 42b (13s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 02:16:57 (1781590617) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (13s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 02:17:11 (1781590631) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 02:17:26 (1781590646) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 02:17:39 (1781590659) Conflict PASS 42f (14s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 02:17:53 (1781590673) Conflict PASS 42g (13s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 02:18:06 (1781590686) Conflict PASS 42h (12s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 02:18:18 (1781590698) PASS 43a (82s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 02:19:40 (1781590780) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 02:19:54 (1781590794) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 02:20:10 (1781590810) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (13s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 02:20:23 (1781590823) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 02:20:38 (1781590838) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (13s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 02:20:51 (1781590851) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 02:21:05 (1781590865) Conflict PASS 43h (14s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 02:21:19 (1781590879) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 02:21:23 (1781590883) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (115s) == sanityn test 43k: unlink vs create ==================== 02:23:17 (1781590997) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1121s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 02:41:58 (1781592118) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Jun 16 02:42:06 EDT 2026 PASS 44a (14s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 02:42:12 (1781592132) open: File exists Conflict PASS 44b (14s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 02:42:27 (1781592147) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 02:42:42 (1781592162) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 02:42:58 (1781592178) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 02:43:12 (1781592192) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 02:43:27 (1781592207) Conflict PASS 44g (12s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 02:43:39 (1781592219) Conflict PASS 44h (13s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 02:43:52 (1781592232) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 02:43:55 (1781592235) PASS 45a (143s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 02:46:18 (1781592378) Conflict PASS 45b (16s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 02:46:34 (1781592394) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 02:46:49 (1781592409) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (12s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 02:47:01 (1781592421) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 02:47:16 (1781592436) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 02:47:29 (1781592449) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (15s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 02:47:45 (1781592465) Conflict PASS 45h (15s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 02:48:00 (1781592480) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 02:48:04 (1781592484) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1138s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 03:07:02 (1781593622) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 03:07:15 (1781593635) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 03:07:29 (1781593649) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 03:07:42 (1781593662) Conflict PASS 46d (13s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 03:07:56 (1781593676) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 03:08:09 (1781593689) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 03:08:21 (1781593701) Conflict PASS 46g (14s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 03:08:35 (1781593715) Conflict PASS 46h (16s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 03:08:51 (1781593731) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 03:08:55 (1781593735) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 03:08:59 (1781593739) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (5s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 03:09:04 (1781593744) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (7s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 03:09:11 (1781593751) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (6s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 03:09:17 (1781593757) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (5s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 03:09:22 (1781593762) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (5s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 03:09:27 (1781593767) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (5s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 03:09:33 (1781593773) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.106118 s, 96.5 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 03:09:46 (1781593786) 0+1 records in 0+1 records out 158 bytes copied, 0.050237 s, 3.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217368) - No such process PASS 51a (12s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 03:09:58 (1781593798) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0130252 s, 78.6 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841735 s, 12.5 MB/s 3145728 PASS 51b (22s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 03:10:21 (1781593821) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0265767 s, 38.5 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 03:10:35 (1781593835) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098418 s, 10.7 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (10s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 03:10:45 (1781593845) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2452 0x994 0x240000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 03:10:56 (1781593856) llite.lustre-ffff93d0c2feb000.xattr_cache=1 llite.lustre-ffff93d0c5589800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:10:58.000000000 -0400 Modify: 2026-06-16 03:10:58.000000000 -0400 Change: 2026-06-16 03:10:58.000000000 -0400 Birth: 2026-06-16 03:10:58.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:11:04.000000000 -0400 Modify: 2026-06-16 03:11:05.000000000 -0400 Change: 2026-06-16 03:11:05.000000000 -0400 Birth: 2026-06-16 03:11:05.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:11:11.000000000 -0400 Modify: 2026-06-16 03:11:11.000000000 -0400 Change: 2026-06-16 03:11:11.000000000 -0400 Birth: 2026-06-16 03:11:11.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:11:11.000000000 -0400 Modify: 2026-06-16 03:11:12.000000000 -0400 Change: 2026-06-16 03:11:12.000000000 -0400 Birth: 2026-06-16 03:11:11.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:11:17.000000000 -0400 Modify: 2026-06-16 03:11:18.000000000 -0400 Change: 2026-06-16 03:11:18.000000000 -0400 Birth: 2026-06-16 03:11:18.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:11:17.000000000 -0400 Modify: 2026-06-16 03:11:18.000000000 -0400 Change: 2026-06-16 03:11:18.000000000 -0400 Birth: 2026-06-16 03:11:18.000000000 -0400 PASS 54 (34s) == sanityn test 55a: rename vs unlink target dir ========= 03:11:30 (1781593890) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (16s) == sanityn test 55b: rename vs unlink source dir ========= 03:11:46 (1781593906) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (16s) == sanityn test 55c: rename vs unlink orphan target dir == 03:12:02 (1781593922) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (20s) == sanityn test 55d: rename file vs link ================= 03:12:22 (1781593942) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 03:12:38 (1781593958) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 03:12:42 (1781593962) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (23s) == sanityn test 55g: rename: race with trylock =========== 03:13:06 (1781593986) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (27s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 03:13:33 (1781594013) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1781594015 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.476868 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (137s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 03:15:51 (1781594151) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1781594157 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.797871 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (170s) == sanityn test 60: Verify data_version behaviour ======== 03:18:41 (1781594321) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 03:18:52 (1781594332) PASS 70a (12s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 03:19:04 (1781594344) PASS 70b (10s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 03:19:14 (1781594354) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0163411 s, 4.0 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 03:19:18 (1781594358) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0366023 s, 1.8 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 03:19:23 (1781594363) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 03:19:28 (1781594368) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0343685 s, 1.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 03:19:32 (1781594372) llite.lustre-ffff93d0c2feb000.xattr_cache=1 llite.lustre-ffff93d0c5589800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (10s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 03:19:42 (1781594382) llite.lustre-ffff93d0c2feb000.xattr_cache=1 llite.lustre-ffff93d0c5589800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 03:19:51 (1781594391) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0236032 s, 43.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0395363 s, 25.9 kB/s 230210: get lock2 230212: get lock1 230210: done 230210: get lock1 230212: done 230212: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 230212: cannot get lock2: Resource deadlock avoided 230212: exit rc=0 230210: done 230210: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 03:20:03 (1781594403) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0880385 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514451 s, 20.4 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230794 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (16s) == sanityn test 76: Verify MDT open_files listing ======== 03:20:19 (1781594419) opening files: ........ closing files: ........ PASS 76 (106s) == sanityn test 77a: check FIFO NRS policy =============== 03:22:05 (1781594525) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.599543 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862512 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870984 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860931 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161669 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198854 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134591 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169735 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193923 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21414 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204088 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212125 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264012 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301239 s, 3.5 MB/sldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235921 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328064 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30277 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104185 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136583 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844218 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00661695 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143828 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205916 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194879 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00665177 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177957 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899979 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083205 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774059 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740517 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646764 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695079 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0552418 s, 19.0 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (12s) == sanityn test 77b: check CRR-N NRS policy ============== 03:22:17 (1781594537) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663585 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655502 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876363 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102247 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176374 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132861 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247274 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168825 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18432 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287193 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247884 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322992 s, 3.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29848 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370426 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35673 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380882 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325228 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605567 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109829 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123545 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745897 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987082 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960615 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110875 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122128 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875889 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929505 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114915 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957917 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926045 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108331 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14307 s, 7.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145753 s, 7.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.569979 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103193 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149944 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125049 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201371 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158762 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128783 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772821 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213991 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146354 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338599 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338018 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391611 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250817 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404323 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300632 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465108 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00607678 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218397 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201973 s, 51.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026399 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00541587 s, 194 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867211 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132869 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130214 s, 80.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116184 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124555 s, 84.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196721 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310108 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142234 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127993 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795982 s, 13.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105479 s, 9.9 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (18s) == sanityn test 77c: check ORR NRS policy ================ 03:22:35 (1781594555) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.583271 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145193 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15405 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204253 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245259 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240449 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230997 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332178 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254683 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360043 s, 2.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344343 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337537 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303221 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364286 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327514 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365679 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361739 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063825 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165904 s, 63.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931637 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211063 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536278 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585326 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941195 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06999 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647041 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00859005 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240346 s, 43.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146783 s, 7.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1493 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00763615 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104282 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0238909 s, 43.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.499887 s, 33.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517425 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489563 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824393 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156458 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218046 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210615 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295368 s, 3.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272479 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33487 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34054 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316698 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298888 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342926 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31399 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364347 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52456 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017291 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830117 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101493 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11032 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694308 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511498 s, 20.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115469 s, 9.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576567 s, 18.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786902 s, 13.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112723 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0874149 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774914 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624846 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899364 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165564 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824231 s, 12.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.653994 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831779 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140515 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140738 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181478 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923329 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226248 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348812 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242323 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40168 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310586 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293464 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353279 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275486 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400821 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14921 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443678 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800138 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188555 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749158 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740965 s, 14.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136355 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135529 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693406 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103263 s, 10.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0123363 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147849 s, 70.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189706 s, 55.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153701 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114582 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144204 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00949647 s, 110 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229638 s, 45.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (23s) == sanityn test 77d: check TRR nrs policy ================ 03:22:59 (1781594579) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.623897 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745242 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13531 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196635 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116613 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163962 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0618935 s, 16.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21619 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270322 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172999 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189702 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242111 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278726 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236863 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185323 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201384 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202196 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681105 s, 15.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820918 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717737 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821455 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130157 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131164 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842928 s, 12.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088317 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906241 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10848 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068176 s, 15.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099886 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957196 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808308 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770702 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969193 s, 10.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.553502 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0838995 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106122 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157534 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161765 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1778 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194173 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167345 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160848 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298081 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33067 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379527 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358926 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374804 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381173 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348887 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363756 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477215 s, 22.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146525 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611672 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953159 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117017 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061442 s, 17.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877842 s, 11.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949685 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00905992 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986879 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117039 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115923 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134621 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168168 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465762 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208067 s, 5.0 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.595136 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140072 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134006 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109296 s, 9.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193574 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287328 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184299 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243181 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298054 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329471 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320216 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284354 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233052 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233876 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183151 s, 5.7 MB/sldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.43635 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269399 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990711 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10129 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122393 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107234 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127328 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115182 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10338 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794614 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119369 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160718 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882132 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107185 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06411 s, 16.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104572 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161864 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11765 s, 8.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (22s) == sanityn test 77e: check TBF NID nrs policy ============ 03:23:20 (1781594600) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.44@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.624013 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148453 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156399 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216909 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112384 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190612 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129149 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244985 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307492 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287661 s, 3.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19886 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265743 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449408 s, 2.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370658 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416326 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.491291 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410684 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673498 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280445 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110103 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409736 s, 25.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108998 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10038 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604008 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574538 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533487 s, 19.7 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129782 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110446 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10574 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0884374 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991841 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114069 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925975 s, 11.3 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.663014 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152044 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16745 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156787 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248057 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359753 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390559 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403992 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367348 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387243 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365649 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.478709 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.438152 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.584933 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.55438 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.569763 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00921773 s, 114 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125039 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063801 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663055 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134705 s, 7.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132259 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138119 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154025 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159699 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253368 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16671 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150513 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765488 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.713097 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161379 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167639 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204553 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.652375 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865139 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120719 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210874 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214737 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278456 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290104 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371837 s, 2.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198146 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375006 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191311 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285963 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387961 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314437 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.437947 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37665 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.393 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00731648 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062672 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631992 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620923 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119458 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124735 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828796 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143878 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107251 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100253 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146977 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18282 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155169 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812229 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175131 s, 6.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.553355 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582368 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876732 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573196 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640248 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188712 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485884 s, 21.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202646 s, 5.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132954 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621117 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170257 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15406 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317092 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287117 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293701 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374706 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295834 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886656 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690678 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796967 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800028 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800718 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741318 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858425 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866337 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143077 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919887 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911438 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106838 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976461 s, 10.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969602 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0945042 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100224 s, 10.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (31s) == sanityn test 77f: check TBF JobID nrs policy ========== 03:23:52 (1781594632) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.880581 s, 19.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139658 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681411 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223188 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26783 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290195 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318943 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270141 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31855 s, 3.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368712 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277659 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316114 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293669 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.407023 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289012 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317923 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383804 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00810045 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106199 s, 98.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947611 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664291 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211702 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805501 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152154 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123832 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871682 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129818 s, 8.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153854 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230703 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13748 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137127 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196425 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589399 s, 17.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.762655 s, 22.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877813 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990486 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17559 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168868 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187247 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100412 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239204 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232515 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207906 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251581 s, 4.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297224 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246262 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283695 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303657 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360433 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375021 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812877 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097218 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796513 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102971 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153708 s, 6.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100232 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887529 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149843 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130929 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124062 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977077 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144358 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148837 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135618 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14351 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126729 s, 8.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.67316 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06575 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103233 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270658 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237165 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267888 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23165 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196503 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316249 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279805 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292504 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265226 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307199 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395696 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307643 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30758 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461387 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0061976 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648984 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521385 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115388 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428883 s, 24.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372089 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00726132 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990641 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108721 s, 96.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106918 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0828301 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474651 s, 22.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174803 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756383 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104463 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167797 s, 62.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.740126 s, 22.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107148 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162326 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163562 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128635 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130115 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17556 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612362 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648147 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23708 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154195 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35613 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176984 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227585 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229307 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323159 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236387 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154766 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054149 s, 19.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378415 s, 27.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189861 s, 55.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00263383 s, 398 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394961 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234398 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190268 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264351 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168172 s, 62.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925954 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243905 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198835 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183873 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964912 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533973 s, 19.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (30s) == sanityn test 77g: Change TBF type directly ============ 03:24:22 (1781594662) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (14s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 03:24:36 (1781594676) oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg644-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg644-client: oleg644-server: ssh exited with exit code 2 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg644-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg644-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg644-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 PASS 77h (15s) == sanityn test 77i: Change rank of TBF rule ============= 03:24:52 (1781594692) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (28s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 03:25:20 (1781594720) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527343 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777986 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113777 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248792 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291325 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289276 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280747 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327214 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340126 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358731 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337588 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346474 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289478 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341499 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320954 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389638 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350524 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564981 s, 18.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651086 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779408 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584166 s, 17.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462333 s, 22.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597867 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637704 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.065565 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708204 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597785 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967359 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737305 s, 14.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725915 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716488 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612289 s, 17.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081368 s, 12.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.5823 s, 13.8 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.701 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (68s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 03:26:28 (1781594788) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3236 s, 12.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685392 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084625 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139911 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150374 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12831 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909749 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995749 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160201 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126362 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144085 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238007 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231179 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336778 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321221 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309658 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246494 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061191 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894371 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815328 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958339 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104109 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122596 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117267 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111949 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117563 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111783 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105299 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139742 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12156 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139411 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137104 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13912 s, 7.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9125 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9531 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3029 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871722 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768652 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155597 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154277 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104168 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178768 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15598 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204811 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18967 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29558 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302979 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334769 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234519 s, 4.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31869 s, 3.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327405 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295544 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475591 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766214 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737549 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543581 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587501 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969744 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733168 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783988 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964122 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779521 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115537 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0957081 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922273 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872397 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101453 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104375 s, 10.0 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9779 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8026 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (151s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 03:28:59 (1781594939) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33794 s, 12.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136422 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872876 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150891 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240291 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163689 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448695 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285014 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770742 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254793 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341248 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341731 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315052 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22953 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412162 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286346 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.546603 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667562 s, 15.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533888 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490202 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361861 s, 29.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090105 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402116 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925448 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708596 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103047 s, 10.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608644 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703251 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641105 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933242 s, 11.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786589 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925733 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904817 s, 11.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0841 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9846 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30458 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781146 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175351 s, 59.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086475 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126005 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110268 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192387 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145613 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178996 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174173 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197894 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105819 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176582 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099549 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211452 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205839 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250966 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857345 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935647 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930668 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12857 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091496 s, 11.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103887 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131768 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12527 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905264 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135081 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117663 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124125 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170839 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169654 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134121 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146806 s, 7.1 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9424 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7678 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (161s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 03:31:40 (1781595100) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.687682 s, 24.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914758 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142674 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172744 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316427 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333973 s, 3.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335043 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403912 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331376 s, 3.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319072 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48224 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.4083 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536944 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433702 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.55341 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.545846 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.555615 s, 1.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=6 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025783 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137367 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0139299 s, 75.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915849 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129213 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114322 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815178 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119396 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112664 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149327 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116065 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120948 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187854 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21232 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212455 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169112 s, 62.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.38792 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1002 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.44@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.44@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.645839 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11179 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107685 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14547 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210655 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197025 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293235 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242686 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243304 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327323 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283016 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41319 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401869 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420862 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344729 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391477 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403257 s, 2.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123162 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120491 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135177 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179135 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124953 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180881 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141091 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148019 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147178 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142187 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144057 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156304 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150312 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155686 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159992 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176585 s, 5.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89156 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0555 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.44@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.10305 s, 15.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155675 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795902 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245358 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113696 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227916 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121952 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121106 s, 8.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229869 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390208 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40395 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353897 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.582562 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.487247 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323451 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.610257 s, 1.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.62651 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112841 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196648 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944698 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134647 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841477 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120017 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143883 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152213 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137618 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120097 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143713 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169378 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181867 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178744 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204998 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222638 s, 4.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.5636 s, 9.9 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.41865 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.535979 s, 31.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753412 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698048 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973721 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839198 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939657 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858054 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374895 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403266 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424025 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468534 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447926 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408654 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.457454 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494742 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.496029 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512249 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100038 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395928 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135432 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0637513 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734697 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113692 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216517 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112872 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942387 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152458 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116373 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691531 s, 15.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126118 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11703 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133773 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131624 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4931 s, 8.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0054 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.69465 s, 13.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95072 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613612 s, 27.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101107 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947345 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281255 s, 3.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326212 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212726 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236542 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273078 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358233 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278344 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35004 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375134 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327436 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344009 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362801 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313949 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.39762 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419649 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050083 s, 20.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115609 s, 9.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054781 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858277 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659867 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778767 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898184 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105083 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805929 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870245 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113822 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918492 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124892 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142053 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150016 s, 7.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0023 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.5768 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.864894 s, 19.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071087 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107045 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146255 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138914 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166879 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202478 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308777 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268395 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285516 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.448248 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470653 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.507679 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518167 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494662 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.485627 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536199 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371386 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909845 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161817 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182865 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106982 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19975 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159188 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870011 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127116 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814671 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070308 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623491 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567554 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510247 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406626 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318214 s, 33.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.95701 s, 11.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0313 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.717216 s, 23.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760015 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697459 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102719 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747351 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110427 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674706 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608819 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561684 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120538 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780344 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235211 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260625 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26973 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362953 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276195 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376746 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387191 s, 27.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434686 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673477 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00838391 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00769149 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101337 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0115286 s, 91.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0201387 s, 52.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113469 s, 92.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970203 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794807 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980665 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832841 s, 12.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103239 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232009 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142668 s, 73.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4254 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0519 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8947 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0409 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (468s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 03:39:29 (1781595569) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.778024 s, 21.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149575 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144789 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166364 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219735 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262757 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319849 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349187 s, 3.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376404 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340557 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126943 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365359 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.419405 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483801 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477606 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416744 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.492744 s, 2.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687364 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234359 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789153 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379022 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0863754 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086233 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573736 s, 18.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810077 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502419 s, 20.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132765 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151063 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562095 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144162 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069849 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154286 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876802 s, 12.0 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89618 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0521 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 03:40:25 (1781595625) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.44@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.44@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00994 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129561 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164199 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252395 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172937 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249639 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181747 s, 5.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31836 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392425 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.35972 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344772 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411543 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131762 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45368 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.369758 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468631 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471353 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206342 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213245 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176429 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18975 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113263 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181335 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115401 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143614 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141238 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123251 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152704 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126692 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126879 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133343 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101484 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164874 s, 6.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=2 removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.231 s, 10.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96307 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (61s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 03:41:26 (1781595686) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.44@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.563223 s, 29.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125933 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197445 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137956 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749879 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281619 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181111 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220216 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235665 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177784 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373706 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.399093 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42627 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344442 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.409941 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459156 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512982 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789214 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490135 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119325 s, 8.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741787 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106023 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075554 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0899866 s, 11.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668995 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138625 s, 7.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100799 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112093 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14948 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107695 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10623 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110972 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129537 s, 8.1 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.8613 s, 13.3 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09692 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (51s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 03:42:17 (1781595737) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.825623 s, 20.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141828 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219555 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310544 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137565 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309194 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354776 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301392 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342626 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421451 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297484 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392229 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365305 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447901 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202776 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.503948 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217007 s, 4.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237152 s, 44.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237358 s, 44.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624791 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208292 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0456309 s, 23.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791492 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00993 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844677 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069695 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082504 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114841 s, 9.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0153345 s, 68.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036945 s, 28.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906587 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091631 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120016 s, 8.7 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.8154 s, 8.9 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.96296 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82558 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90149 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (104s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 03:44:02 (1781595842) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03634 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726709 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896892 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766758 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165137 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150177 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122689 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112324 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194974 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1748 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242841 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117645 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217447 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221136 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292463 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269632 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291158 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057058 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652638 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361786 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321327 s, 32.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549907 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490355 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294892 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438165 s, 23.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286322 s, 36.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00888514 s, 118 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00732079 s, 143 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602971 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781083 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00460813 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488422 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184066 s, 57.0 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7808 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9572 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (80s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 03:45:22 (1781595922) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.658128 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752681 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099627 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197987 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151654 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177804 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178753 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210559 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246286 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219269 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267867 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291812 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282319 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229179 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284593 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256428 s, 4.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0c2feb000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff93d0c2feb000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275705 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645103 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858951 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528902 s, 19.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.085463 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807478 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900334 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08776 s, 11.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101046 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870768 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.091974 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795753 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102725 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902116 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940704 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107706 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141578 s, 7.4 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.26916 s, 14.4 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91592 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg644-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.621112 s, 27.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04672 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829527 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163811 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207802 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293791 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313579 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267579 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313141 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26511 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308977 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310412 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315137 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278179 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297937 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34043 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347749 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463193 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512758 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597452 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048964 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0615029 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654246 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952086 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669728 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970061 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0754221 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610935 s, 17.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668211 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799326 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781758 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720477 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886687 s, 11.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1822 s, 10.3 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92942 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7822 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97277 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (146s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 03:47:48 (1781596068) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg644-client.virtnet /mnt/lustre (opts:) Stopping client oleg644-client.virtnet /mnt/lustre opts: Stopping clients: oleg644-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg644-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg644-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg644-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg644-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,relatime) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg644-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.44,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg644-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.44,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg644-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.44,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/65245/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.44,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg644-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg644-server' oleg644-server: oleg644-server.virtnet: executing load_modules_local oleg644-server: Loading modules from /home/green/git/lustre-release/lustre oleg644-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg644-server: detected 4 online CPUs by sysfs oleg644-server: MODOPTS_LIBCFS= oleg644-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg644-server: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg644-server: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg644-server: oleg644-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg644-client: oleg644-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre Starting client oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre Started clients oleg644-client.virtnet: 192.168.206.144@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 Starting client oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 Started clients oleg644-client.virtnet: 192.168.206.144@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93d0d6686800.idle_timeout=debug osc.lustre-OST0000-osc-ffff93d0d66d6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0d6686800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0d66d6800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.872658 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487459 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423337 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809501 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164823 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130354 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174567 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180124 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2013 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183514 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206889 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197223 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213254 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246683 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245256 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243402 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23783 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0d66d6800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff93d0d66d6800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500901 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513278 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498578 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498105 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.50767 s, 2.1 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508024 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.525159 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510113 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.506739 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508242 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.514752 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.508876 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.51299 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518855 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529062 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560339 s, 1.9 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.733 s, 7.6 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.33742 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.597627 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622068 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110095 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136657 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148629 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196803 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185489 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21167 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163243 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174349 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283964 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23871 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26343 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298203 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245752 s, 4.3 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff93d0d66d6800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff93d0d66d6800.lock_unused_count=4 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267925 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282011 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189772 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242225 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151496 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161745 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160933 s, 6.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162362 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206427 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175904 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185839 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273447 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233196 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198268 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20812 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210212 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251062 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216473 s, 4.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0d66d6800.lock_unused_count=2 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.41799 s, 14.1 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88273 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (220s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 03:51:29 (1781596289) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg644-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (21s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 03:51:49 (1781596309) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 03:52:00 (1781596320) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg644-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.90853 s, 268 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg644-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0786 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 03:52:59 (1781596379) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.421819 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105947 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121313 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155202 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215017 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176885 s, 5.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276007 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212847 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197016 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300857 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250547 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0d66d6800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff93d0d66d6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337305 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284359 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422103 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278616 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255671 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.426156 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0131733 s, 79.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133462 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126236 s, 83.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164432 s, 63.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385736 s, 272 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00619472 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00930223 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105931 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215169 s, 48.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873131 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117541 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158801 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12651 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433343 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132848 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371215 s, 28.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg644-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg644-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.6144 s, 12.2 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.17541 s, 20.3 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.613668 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960176 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108758 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138759 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116489 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107952 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171406 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132312 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154208 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158035 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163205 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18835 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206163 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269879 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93d0d66d6800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff93d0d66d6800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294749 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295767 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280678 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385785 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425464 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397873 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461625 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.416277 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408118 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465729 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436164 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462625 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452148 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429144 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.449299 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468373 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471848 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.467435 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45855 s, 2.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg644-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg644-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.34895 s, 14.3 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.05652 s, 20.7 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (89s) == sanityn test 77o: Changing rank should not panic ====== 03:54:28 (1781596468) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (13s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 03:54:41 (1781596481) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (108s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 03:56:29 (1781596589) mds.MDS.mdt.nrs_policies=tbf oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg644-client: oleg644-server: ssh exited with exit code 36 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 oleg644-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg644-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg644-client: oleg644-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (31s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 03:57:00 (1781596620) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 269860 kill md_thread xargs PASS 77r (60s) == sanityn test 77s: Check TBF LRU shrinker ============== 03:58:00 (1781596680) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (76s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 03:59:17 (1781596757) ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 03:59:27 (1781596767) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935652 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 03:59:29.000000000 -0400 Modify: 2026-06-16 03:59:29.000000000 -0400 Change: 2026-06-16 03:59:29.000000000 -0400 Birth: 2026-06-16 03:59:29.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (19s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 03:59:46 (1781596786) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 03:59:50 (1781596790) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 03:59:54 (1781596794) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 03:59:57 (1781596797) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (3s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 04:00:01 (1781596801) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 04:00:04 (1781596804) - open/close 658 (time 1781596819.33 total 10.01 last 65.70) - open/close 1311 (time 1781596829.33 total 20.02 last 65.25) - open/close 1904 (time 1781596839.36 total 30.05 last 59.12) total: 2500 open/close in 39.70 seconds: 62.98 ops/second snapshot_time 1781596947.102224552 secs.nsecs start_time 1781596807.949400069 secs.nsecs elapsed_time 139.152824483 secs.nsecs open 2505 samples [usecs] 145 14877 3782712 9530504530 close 2505 samples [usecs] 178 11573 891494 766005572 mknod 2501 samples [usecs] 575 14790 3324485 7640067675 unlink 2500 samples [usecs] 726 61854 5306207 31552603907 mkdir 2 samples [usecs] 1303 2137 3440 6264578 rmdir 2 samples [usecs] 1092 11699 12791 138059065 rename 2500 samples [usecs] 507 67404 18160466 275297592678 getattr 3763 samples [usecs] 7 10127 238629 316569167 parallel_rename_file 2500 samples [usecs] 507 67404 18160466 275297592678 crossdir_rename 2500 samples [usecs] 507 67404 18160466 275297592678 PASS 81d (148s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 04:02:32 (1781596952) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7596 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7596 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 04:02:40 (1781596960) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (3s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 04:02:44 (1781596964) fail_loc=0x8000060b PASS 84 (13s) == sanityn test 85: Lustre API root cache race =========== 04:02:57 (1781596977) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1781596979 Finishing test test1 at 1781596984 PASS 85 (12s) == sanityn test 90: open/create and unlink striped directory ========================================================== 04:03:09 (1781596989) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (4s) == sanityn test 91: chmod and unlink striped directory === 04:03:13 (1781596993) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (5s) == sanityn test 92: create remote directory under orphan directory ========================================================== 04:03:19 (1781596999) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 04:03:23 (1781597003) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5800 0x16a8 0x240000400 1 5763 0x1683 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5801 0x16a9 0x240000400 1 5764 0x1684 0x280000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 04:03:40 (1781597020) osc.lustre-OST0000-osc-ffff93d0d6686800.idle_timeout=0 osc.lustre-OST0000-osc-ffff93d0d66d6800.idle_timeout=0 osc.lustre-OST0001-osc-ffff93d0d6686800.idle_timeout=0 osc.lustre-OST0001-osc-ffff93d0d66d6800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.180191 s, 45.5 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.98612 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 286627 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0393429 s, 104 kB/s osc.lustre-OST0000-osc-ffff93d0d6686800.idle_timeout=debug osc.lustre-OST0000-osc-ffff93d0d66d6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0d6686800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93d0d66d6800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 04:03:55 (1781597035) llite.lustre-ffff93d0d6686800.fast_read=0 llite.lustre-ffff93d0d66d6800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.071833 s, 29.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff93d0d6686800.fast_read=1 llite.lustre-ffff93d0d66d6800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 04:04:15 (1781597055) llite.lustre-ffff93d0d6686800.fast_read=0 llite.lustre-ffff93d0d66d6800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0120287 s, 1.0 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff93d0d6686800.fast_read=1 llite.lustre-ffff93d0d66d6800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 04:04:28 (1781597068) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 04:04:32 (1781597072) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00612833 s, 668 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (6s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 04:04:38 (1781597078) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120286 s, 17.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 04:04:46 (1781597086) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162367 s, 12.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (7s) == sanityn test 100e: DoM: read on open and file size ==== 04:04:53 (1781597093) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0125077 s, 959 kB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 04:05:01 (1781597101) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0173051 s, 237 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 04:05:09 (1781597109) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00905733 s, 452 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 04:05:17 (1781597117) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0108362 s, 378 kB/s PASS 101c (12s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 04:05:32 (1781597132) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1781597135 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1781597139 Finishing test test23 at 1781597139 PASS 103 (17s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 04:05:48 (1781597148) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 04:05:52 (1781597152) fail_loc=0x416 PASS 105 (23s) == sanityn test 106a: Verify the btime via statx() ======= 04:06:15 (1781597175) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 04:06:19 (1781597179) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0386794 s, 26.5 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938187 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 04:06:21.000000000 -0400 Modify: 2026-06-16 04:06:21.000000000 -0400 Change: 2026-06-16 04:06:21.000000000 -0400 Birth: 2026-06-16 04:06:22.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938187 -rw-r--r-- 1 0 0 1781597182 1781597181 1781597181 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938187 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 04:06:21.000000000 -0400 Modify: 2026-06-16 04:06:21.000000000 -0400 Change: 2026-06-16 04:06:21.000000000 -0400 Birth: 2026-06-16 04:06:22.000000000 -0400 1781597181 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 04:06:28 (1781597188) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 04:06:36 (1781597196) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.385705 s, 27.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5770 0x168a 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 04:06:47 (1781597207) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.280927 s, 37.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5810 0x16b2 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7596 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 04:07:02 (1781597222) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0116224 s, 860 kB/s llite.lustre-ffff93d0d6686800.hybrid_io=0 llite.lustre-ffff93d0d66d6800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.56375 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 04:07:15 (1781597235) 192.168.206.144@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg644-client.virtnet /mnt/lustre (opts:) 192.168.206.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg644-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 192.168.206.144@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.144@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg644-client.virtnet /mnt/lustre (opts:) Stopping client oleg644-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg644-server' oleg644-server: oleg644-server.virtnet: executing load_modules_local oleg644-server: Loading modules from /home/green/git/lustre-release/lustre oleg644-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg644-server: detected 4 online CPUs by sysfs oleg644-server: MODOPTS_LIBCFS= oleg644-server: Force libcfs to create 2 CPU partitions Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre Starting client: oleg644-client.virtnet: -o user_xattr,flock 192.168.206.144@tcp:/lustre /mnt/lustre2 PASS 109 (48s) == sanityn test 110: do not grant another lock on resend ========================================================== 04:08:04 (1781597284) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-16 04:08:06.000000000 -0400 Modify: 2026-06-16 04:08:06.000000000 -0400 Change: 2026-06-16 04:08:06.000000000 -0400 Birth: 2026-06-16 04:08:06.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg644-client.virtnet oleg644-client.virtnet PASS 110 (169s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 04:10:53 (1781597453) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (4s) == sanityn test 112: update max-inherit in default LMV === 04:10:57 (1781597457) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 04:11:01 (1781597461) lustre-OST0000-osc-ffff93d0d1dcf000 active. lustre-OST0001-osc-ffff93d0d1dcf000 active. lustre-OST0000-osc-ffff93d0d1dcf000 active. lustre-OST0001-osc-ffff93d0d1dcf000 active. MGC192.168.206.144@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 04:11:08 (1781597468) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 04:11:12 (1781597472) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 04:11:15 (1781597475) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 04:11:19 (1781597479) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0121952 s, 1.7 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.131932 s, 155 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 04:11:28 (1781597488) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.111 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1616 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1231 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1053 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1855 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1936 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1684 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0409 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0756 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.087 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0596 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.143 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1121 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2125 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2326 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3495 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.238 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2742 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg644-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93d0c558f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93d0c558f000.ost_server_uuid in IDLE state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 9889 sec ============== 04:12:51 (1781597571) === sanityn: start cleanup 04:12:53 (1781597573) === Stopping clients: oleg644-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg644-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 04:17:11 (1781597831) ===