-----============= acceptance-small: sanityn ============----- Wed Sep 30 12:28:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg319-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 12:28:28 (1790785708) === Starting client oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Mount client oleg319-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Started clients oleg319-client.virtnet: 192.168.203.119@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg319-client.virtnet: executing check_config_client /mnt/lustre oleg319-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg319-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9eac1008c800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9eac1223c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac1008c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac1223c000.idle_timeout=debug disable quota as required oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 12:29:06 (1790785746) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056976 s, 18.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7623] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 12:29:09 (1790785749) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 12:29:18 (1790785758) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 12:29:26 (1790785766) -rw-r--r-- 1 root root 0 Sep 30 12:29 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (7s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 12:29:33 (1790785773) -rw-r--r-- 1 root root 0 Sep 30 12:29 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 12:29:40 (1790785780) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 12:29:47 (1790785787) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 12:29:54 (1790785794) total 1 -rwxrwxrwx 1 root root 0 Sep 30 12:29 f2a -rwxrwxrwx 1 root root 0 Sep 30 12:29 f2b total 1 -rwxrwxrwx 1 root root 0 Sep 30 12:29 f2a -rwxrwxrwx 1 root root 0 Sep 30 12:29 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0157327 s, 32.5 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 12:30:01 (1790785801) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (3s) == sanityn test 2g: check blocks update on sync write ==== 12:30:04 (1790785804) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.392479 s, 5.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 12:30:11 (1790785811) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 12:30:18 (1790785818) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 12:30:27 (1790785827) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (7s) == sanityn test 6: remove of open file on other node ============================================================================ 12:30:34 (1790785834) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 12:30:41 (1790785841) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 12:30:48 (1790785848) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 12:30:54 (1790785854) PASS 9a (8s) == sanityn test 9b: append to striped sparse file ======== 12:31:02 (1790785862) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442288 s, 23.7 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0531942 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 12:31:08 (1790785868) 1+0 records in 1+0 records out 1 byte copied, 0.00414155 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0173185 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0154936 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0082809 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00919489 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172834 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0158365 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0085511 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0130147 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0276279 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00895368 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0267126 s, 0.0 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 12:31:17 (1790785877) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0457672 s, 67.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0495381 s, 82.7 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000242013 s, 12.7 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 12:31:23 (1790785883) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 298: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 12:31:31 (1790785891) warning: '-runtime' deprecated, use '-t runtime' instead - link 307 (time 1790785903.75 total 10.02 last 30.65) - link 598 (time 1790785913.75 total 20.02 last 29.09) - link 903 (time 1790785923.78 total 30.04 last 30.43) - link 1173 (time 1790785933.80 total 40.07 last 26.94) - link 1460 (time 1790785943.80 total 50.07 last 28.69) - link 1747 (time 1790785953.82 total 60.09 last 28.65) - link 2054 (time 1790785963.84 total 70.11 last 30.64) - link 2371 (time 1790785973.85 total 80.12 last 31.69) - link 2688 (time 1790785983.89 total 90.16 last 31.57) total: 2988 link in 100.03 seconds: 29.87 ops/second using seed 1893686782 running for 100 seconds - stat 4728 (time 1790785905 ; total 11 ; last 11) - stat 9394 (time 1790785916 ; total 22 ; last 11) - stat 10000 (time 1790785917 ; total 23 ; last 1) - stat 15016 (time 1790785928 ; total 34 ; last 11) - stat 19498 (time 1790785939 ; total 45 ; last 11) - stat 20000 (time 1790785940 ; total 46 ; last 1) - stat 23403 (time 1790785951 ; total 57 ; last 11) - stat 26224 (time 1790785962 ; total 68 ; last 11) - stat 28579 (time 1790785973 ; total 79 ; last 11) - stat 30000 (time 1790785978 ; total 84 ; last 5) - stat 32583 (time 1790785989 ; total 95 ; last 11) total: 33882 stats in 100 seconds: 338.820007 stats/second - unlinked 0 (time 1790785944 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile2988) error: No such file or directory total: 2987 unlinks in 86 seconds: 34.732559 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21425) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21427) - No such process PASS 12 (144s) == sanityn test 13: test directory page revocation ======= 12:33:55 (1790786035) f13 f13 PASS 13 (11s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 12:34:06 (1790786046) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 333: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 22611 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 12:34:13 (1790786053) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 23190 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 12:34:20 (1790786060) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 348: 23768 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 12:34:27 (1790786067) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 360: 24351 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 12:34:34 (1790786074) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 371: 24933 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 12:34:43 (1790786083) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (52s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 12:35:35 (1790786135) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.462549 s, 18.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.291547 s, 28.8 MB/s Chance of close/open is 1 in 50 Seed set to 6140 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790786141.499187 mapwrite 0x0029af thru 0x00f08e (0xc6e0 bytes) 000003[0] 1790786141.688987 punch from 0x003f51 to 0x0080d6 (0x4186 bytes) 000004[1] 1790786141.762245 punch from 0x009f32 to 0x00f08e (0x515d bytes) 000005[1] 1790786141.832799 mapwrite 0x0a889a thru 0x0b7cb1 (0xf418 bytes) 000009[0] 1790786141.995133 punch from 0x088d11 to 0x08d721 (0x4a11 bytes) 000010[1] 1790786142.032919 punch from 0x00c19d to 0x016c93 (0xaaf7 bytes) 000011[1] 1790786142.076046 write 0x7e7788 thru 0x7ea7d5 (0x304e bytes) 000017[1] 1790786142.321713 mapread 0x62ec3f thru 0x636041 (0x7403 bytes) 000018[0] 1790786142.404906 read 0x2982ba thru 0x2a77fe (0xf545 bytes) 000019[1] 1790786142.590673 mapwrite 0x90230e thru 0x906bfb (0x48ee bytes) 000021[1] 1790786142.807541 mapwrite 0x4ad148 thru 0x4baadf (0xd998 bytes) truncating to largest ever: 0x8bdcd5 000022[1] 1790786142.987719 trunc from 0x906bfc to 0x8bdcd4 (0x48f27 bytes) 000024[0] 1790786143.078640 punch from 0x5a8d25 to 0x5b468b (0xb967 bytes) 000027[1] 1790786143.238577 write 0x5e4a08 thru 0x5eca88 (0x8081 bytes) truncating to largest ever: 0x8e6567 000029[1] 1790786143.328741 trunc from 0x8bdcd5 to 0x8e6566 (0x28892 bytes) 000033[0] 1790786143.407899 mapwrite 0x77e18b thru 0x787502 (0x9378 bytes) 000034[1] 1790786143.538971 read 0x844f41 thru 0x850334 (0xb3f4 bytes) 000036[0] 1790786143.576638 read 0x860013 thru 0x86ef33 (0xef21 bytes) 000037[1] 1790786143.619909 mapread 0x12eee3 thru 0x130b19 (0x1c37 bytes) 000039[0] 1790786143.698421 mapread 0x05771a thru 0x062478 (0xad5f bytes) 000040[0] 1790786143.719457 mapread 0x1cde4b thru 0x1d0251 (0x2407 bytes) 000041[0] 1790786143.732553 mapread 0x83bba3 thru 0x849fb9 (0xe417 bytes) 000042[1] 1790786143.768120 read 0x0a0374 thru 0x0a5b44 (0x57d1 bytes) 000043[0] 1790786143.822654 mapwrite 0x74313c thru 0x746ead (0x3d72 bytes) 000046[0] 1790786144.033246 write 0x0e6380 thru 0x0edc77 (0x78f8 bytes) 000047[1] 1790786144.062114 trunc from 0x8e6567 to 0x6831b9 (0x2633ad bytes) 000049[0] 1790786144.152890 mapwrite 0x7c6a05 thru 0x7cf564 (0x8b60 bytes) 000051[0] 1790786144.288550 mapread 0x519968 thru 0x51dafb (0x4194 bytes) 000054[0] 1790786144.358743 punch from 0x622dcf to 0x62d766 (0xa998 bytes) 000056[1] 1790786144.366186 punch from 0x16e823 to 0x174d06 (0x64e4 bytes) 000057[1] 1790786144.408831 punch from 0x0e0fab to 0x0efc54 (0xecaa bytes) 000058[0] 1790786144.442369 read 0x7aa366 thru 0x7b6694 (0xc32f bytes) 000059[1] 1790786144.482166 write 0x653ee0 thru 0x657284 (0x33a5 bytes) 000061[0] 1790786144.547174 mapread 0x4b01c9 thru 0x4b8e66 (0x8c9e bytes) 000062[0] 1790786144.634881 mapread 0x217e91 thru 0x21ca8b (0x4bfb bytes) 000063[0] 1790786144.682228 trunc from 0x7cf565 to 0x4543eb (0x37b179 bytes) 000064[0] 1790786144.739635 write 0x380dde thru 0x38c985 (0xbba8 bytes) 000065[0] 1790786144.754881 trunc from 0x4543ec to 0x21601e (0x23e3cd bytes) 000066[0] 1790786144.788431 write 0x78261b thru 0x792261 (0xfc47 bytes) 000067[0] 1790786144.799227 read 0x0de441 thru 0x0ecf1f (0xeadf bytes) 000068[0] 1790786144.830695 read 0x22c19d thru 0x23173d (0x55a1 bytes) 000069[1] 1790786144.884178 write 0x5e24a8 thru 0x5e72f5 (0x4e4e bytes) 000071[0] 1790786145.011792 mapread 0x6627fa thru 0x664cdb (0x24e2 bytes) 000073[0] 1790786145.052864 punch from 0x3a6d6c to 0x3b5371 (0xe606 bytes) 000074[1] 1790786145.076032 mapwrite 0x54293b thru 0x54cc77 (0xa33d bytes) 000075[1] 1790786145.189402 punch from 0x575f10 to 0x585ce4 (0xfdd5 bytes) 000079[0] 1790786145.255979 punch from 0x1d2c2c to 0x1d4537 (0x190c bytes) 000081[0] 1790786145.354640 trunc from 0x792262 to 0x263dce (0x52e493 bytes) 000082[0] 1790786145.451623 mapread 0x19cc2c thru 0x1a70b0 (0xa485 bytes) 000083[0] 1790786145.502014 punch from 0x15c2a0 to 0x1672ba (0xb01b bytes) 000085[0] 1790786145.530398 mapread 0x0cd8d9 thru 0x0d0599 (0x2cc1 bytes) 000086[1] 1790786145.568218 mapwrite 0x053c4c thru 0x060d9c (0xd151 bytes) 000087[0] 1790786145.748041 read 0x0eddf1 thru 0x0f77ab (0x99bb bytes) 000090[1] 1790786145.814639 mapwrite 0x66fc1d thru 0x67b39e (0xb782 bytes) 000092[1] 1790786146.011036 punch from 0x2867df to 0x28cc04 (0x6426 bytes) 000099[1] 1790786146.150584 punch from 0x271699 to 0x27f593 (0xdefb bytes) 000100[1] 1790786146.172756 trunc from 0x67b39f to 0x5739d4 (0x1079ca bytes) 000101[1] 1790786146.232381 write 0x025761 thru 0x026c3b (0x14db bytes) 000102[0] 1790786146.251197 read 0x1bad15 thru 0x1c61be (0xb4aa bytes) 000103[0] 1790786146.294956 trunc from 0x5739d5 to 0x3aa54c (0x1c9488 bytes) 000104[1] 1790786146.361818 punch from 0x19c78d to 0x1a1af8 (0x536c bytes) 000106[0] 1790786146.401334 punch from 0x130c40 to 0x13204b (0x140c bytes) 000109[1] 1790786146.453090 punch from 0x2a2c73 to 0x2a5e82 (0x3210 bytes) 000111[1] 1790786146.499241 punch from 0x0a3efb to 0x0ae620 (0xa726 bytes) 000112[1] 1790786146.521713 write 0x83c34b thru 0x84914d (0xce03 bytes) 000114[1] 1790786146.560364 mapread 0x554eb5 thru 0x55a28e (0x53da bytes) 000115[1] 1790786146.627115 trunc from 0x84914e to 0x073d48 (0x7d5405 bytes) 000116[0] 1790786146.700031 punch from 0x0555d1 to 0x064c77 (0xf6a7 bytes) 000117[1] 1790786146.726968 punch from 0x0409fc to 0x0411ff (0x804 bytes) 000118[0] 1790786146.739133 trunc from 0x073d49 to 0x1d7699 (0x163951 bytes) 000119[1] 1790786146.775725 trunc from 0x1d769a to 0x3d2385 (0x1facec bytes) 000120[0] 1790786146.813183 mapwrite 0x652b34 thru 0x661bfe (0xf0cb bytes) 000121[1] 1790786146.985076 mapread 0x4dc149 thru 0x4e20d8 (0x5f90 bytes) 000123[0] 1790786147.075312 punch from 0x2453f8 to 0x25198a (0xc593 bytes) 000124[1] 1790786147.114280 punch from 0x024a90 to 0x02e487 (0x99f8 bytes) 000125[0] 1790786147.133300 trunc from 0x661bff to 0x83d59d (0x1db99f bytes) 000128[1] 1790786147.289881 mapread 0x2b402b thru 0x2c1aae (0xda84 bytes) 000130[1] 1790786147.357219 mapread 0x7f7001 thru 0x7f943e (0x243e bytes) 000131[1] 1790786147.441261 punch from 0x8291f2 to 0x8342da (0xb0e9 bytes) 000133[0] 1790786147.469425 punch from 0x19435d to 0x1a1f3c (0xdbe0 bytes) 000134[1] 1790786147.497656 mapwrite 0x73c9db thru 0x74b819 (0xee3f bytes) 000137[0] 1790786147.641929 mapwrite 0x9afef7 thru 0x9b1143 (0x124d bytes) 000138[1] 1790786147.749314 mapwrite 0x5a9af5 thru 0x5aceaf (0x33bb bytes) 000140[1] 1790786147.900231 mapread 0x56ada2 thru 0x571652 (0x68b1 bytes) 000142[0] 1790786147.964701 write 0x0fb564 thru 0x0fe909 (0x33a6 bytes) 000143[1] 1790786148.001136 read 0x8f2750 thru 0x8f44b5 (0x1d66 bytes) 000145[0] 1790786148.059815 mapwrite 0x043e7a thru 0x043f2b (0xb2 bytes) 000148[1] 1790786148.188565 write 0x5ad15e thru 0x5b80ff (0xafa2 bytes) 000150[1] 1790786148.232369 punch from 0x2af93c to 0x2b1330 (0x19f5 bytes) 000151[0] 1790786148.260684 trunc from 0x9b1144 to 0x2bc695 (0x6f4aae bytes) 000152[0] 1790786148.346024 mapwrite 0x65556a thru 0x6585bf (0x3056 bytes) 000153[0] 1790786148.471885 write 0x4e9ac4 thru 0x4f87de (0xed1b bytes) 000156[1] 1790786148.548141 write 0x45fea8 thru 0x468855 (0x89ae bytes) 000157[1] 1790786148.604824 read 0x4527b5 thru 0x45dc7b (0xb4c7 bytes) 000158[1] 1790786148.634192 trunc from 0x6585c0 to 0x021a3f (0x636b80 bytes) 000160[1] 1790786148.672962 write 0x8f2546 thru 0x8fd73e (0xb1f9 bytes) 000161[0] 1790786148.678110 mapread 0x52b7dd thru 0x5341ee (0x8a12 bytes) 000162[1] 1790786148.833088 punch from 0x6afa8f to 0x6be5b4 (0xeb26 bytes) 000163[1] 1790786148.845175 write 0x446fff thru 0x455a2e (0xea30 bytes) 000165[1] 1790786148.857042 mapread 0x1afa2c thru 0x1bf92d (0xff02 bytes) 000168[0] 1790786148.879574 mapwrite 0x41eb6b thru 0x4203b9 (0x184f bytes) 000169[1] 1790786149.072899 mapwrite 0x0b0140 thru 0x0ba8b7 (0xa778 bytes) 000170[0] 1790786149.250382 punch from 0x2742ff to 0x2772a2 (0x2fa4 bytes) 000171[0] 1790786149.338090 mapread 0x60ca47 thru 0x618e93 (0xc44d bytes) 000172[1] 1790786149.551953 trunc from 0x8fd73f to 0x0ca458 (0x8332e6 bytes) 000173[1] 1790786149.682795 mapread 0x0182fb thru 0x019555 (0x125b bytes) 000174[0] 1790786149.822396 punch from 0x0ad546 to 0x0b8c47 (0xb702 bytes) 000175[0] 1790786149.921702 trunc from 0x0ca459 to 0x1942ca (0xc9e72 bytes) 000176[0] 1790786150.016085 trunc from 0x1942cb to 0x3b3828 (0x21f55e bytes) 000177[0] 1790786150.077324 read 0x342d6e thru 0x34b349 (0x85dc bytes) 000179[0] 1790786150.250087 punch from 0x1f0494 to 0x1f26d4 (0x2241 bytes) 000184[1] 1790786150.358043 mapwrite 0x16aabd thru 0x171870 (0x6db4 bytes) 000185[0] 1790786150.623338 write 0x041c97 thru 0x044670 (0x29da bytes) 000188[0] 1790786150.703122 write 0x720665 thru 0x73011e (0xfaba bytes) 000189[0] 1790786150.708871 punch from 0x04dacf to 0x051a71 (0x3fa3 bytes) 000190[1] 1790786150.783010 mapread 0x0b9624 thru 0x0c5f92 (0xc96f bytes) 000191[1] 1790786150.851030 punch from 0x4b274e to 0x4bfd28 (0xd5db bytes) 000192[0] 1790786150.889024 mapwrite 0x256426 thru 0x260ffd (0xabd8 bytes) 000195[0] 1790786151.190414 punch from 0x0a098d to 0x0a2355 (0x19c9 bytes) 000197[1] 1790786151.253773 read 0x12b024 thru 0x137353 (0xc330 bytes) 000200[0] 1790786151.329647 mapread 0x2cf2f5 thru 0x2d1653 (0x235f bytes) 000202[0] 1790786151.424028 trunc from 0x73011f to 0x627e6d (0x1082b1 bytes) 000203[0] 1790786151.572450 read 0x1bee85 thru 0x1c51c6 (0x6342 bytes) 000206[1] 1790786151.612077 mapwrite 0x202973 thru 0x20cbfd (0xa28b bytes) 000207[0] 1790786151.862665 mapread 0x0dea87 thru 0x0e6611 (0x7b8b bytes) 000208[1] 1790786151.984953 mapread 0x2dcaf0 thru 0x2e1682 (0x4b93 bytes) 000209[1] 1790786152.103258 read 0x481c48 thru 0x483147 (0x1500 bytes) 000210[0] 1790786152.144567 mapwrite 0x283aa0 thru 0x288d49 (0x52aa bytes) 000211[0] 1790786152.412693 punch from 0x0af08f to 0x0b0bea (0x1b5c bytes) 000212[1] 1790786152.515523 mapwrite 0x63bec6 thru 0x6425b8 (0x66f3 bytes) 000213[0] 1790786152.917336 mapread 0x61e6a3 thru 0x627a7f (0x93dd bytes) 000215[1] 1790786153.015625 punch from 0x606b7d to 0x6158a7 (0xed2b bytes) 000217[1] 1790786153.069451 trunc from 0x6425b9 to 0x74e599 (0x10bfe1 bytes) 000219[0] 1790786153.116189 mapread 0x0a6951 thru 0x0b3e69 (0xd519 bytes) 000220[0] 1790786153.196686 punch from 0x5cc6a3 to 0x5d711a (0xaa78 bytes) 000221[0] 1790786153.221050 punch from 0x0f81ea to 0x107cbf (0xfad6 bytes) 000223[0] 1790786153.301047 trunc from 0x74e59a to 0x83111b (0xe2b82 bytes) 000224[1] 1790786153.354199 read 0x2d75df thru 0x2e2ae6 (0xb508 bytes) 000226[1] 1790786153.402230 mapread 0x3e4a56 thru 0x3e9a4c (0x4ff7 bytes) 000228[1] 1790786153.479077 mapwrite 0x61d7bb thru 0x622fda (0x5820 bytes) truncating to largest ever: 0x9ab14b 000229[1] 1790786153.681753 trunc from 0x83111c to 0x9ab14a (0x17a02f bytes) 000230[1] 1790786153.809692 trunc from 0x9ab14b to 0x92ec7e (0x7c4cc bytes) 000231[1] 1790786153.862908 mapread 0x7ec1b2 thru 0x7fb519 (0xf368 bytes) 000232[1] 1790786153.912192 mapread 0x6b2e0e thru 0x6b8e20 (0x6013 bytes) 000233[0] 1790786153.957277 trunc from 0x92ec7f to 0x9053cb (0x298b3 bytes) 000234[0] 1790786154.081590 read 0x8bce03 thru 0x8bfc7e (0x2e7c bytes) 000235[0] 1790786154.111949 mapwrite 0x2dad13 thru 0x2e8d50 (0xe03e bytes) 000237[0] 1790786154.403959 mapwrite 0x3abade thru 0x3af1f5 (0x3718 bytes) 000238[0] 1790786154.692812 mapread 0x219bcd thru 0x227583 (0xd9b7 bytes) 000240[0] 1790786154.873819 write 0x2a253a thru 0x2abf6b (0x9a32 bytes) 000241[0] 1790786154.953957 punch from 0x30a4f3 to 0x315e67 (0xb975 bytes) 000244[1] 1790786155.031594 write 0x9e83d3 thru 0x9f50a5 (0xccd3 bytes) 000245[1] 1790786155.110709 trunc from 0x9f50a6 to 0x5bd00a (0x43809b bytes) 000246[0] 1790786155.191216 mapread 0x4bbf58 thru 0x4c2107 (0x61b0 bytes) 000247[1] 1790786155.257174 mapread 0x1591a6 thru 0x15e358 (0x51b3 bytes) 000249[1] 1790786155.298697 mapwrite 0x878115 thru 0x87821a (0x106 bytes) 000252[1] 1790786155.468119 mapwrite 0x60cc95 thru 0x6150ae (0x841a bytes) 000253[0] 1790786155.611180 trunc from 0x87821b to 0x8ea4aa (0x72290 bytes) 000257[1] 1790786155.693145 trunc from 0x8ea4ab to 0x27e5cf (0x66bedb bytes) 000258[0] 1790786155.757780 write 0x8f9109 thru 0x8ff534 (0x642c bytes) 000259[0] 1790786155.791797 read 0x1626b6 thru 0x1644a2 (0x1ded bytes) 000260[0] 1790786155.813262 write 0x95d426 thru 0x96c155 (0xed30 bytes) 000261[0] 1790786155.834310 read 0x0c4d03 thru 0x0ce41d (0x971b bytes) 000262[1] 1790786155.857456 read 0x6d92d7 thru 0x6dc583 (0x32ad bytes) 000263[1] 1790786155.902494 punch from 0x24f2cf to 0x25d764 (0xe496 bytes) 000265[0] 1790786155.968423 trunc from 0x96c156 to 0x012e3d (0x959318 bytes) 000266[0] 1790786156.056127 punch from 0x00b12f to 0x012e3d (0x7d0f bytes) 000268[0] 1790786156.093016 trunc from 0x012e3e to 0x47da75 (0x46ac38 bytes) 000271[0] 1790786156.138923 mapwrite 0x8b8bcd thru 0x8bf639 (0x6a6d bytes) 000272[0] 1790786156.356909 mapread 0x27b183 thru 0x286825 (0xb6a3 bytes) 000274[1] 1790786156.548066 trunc from 0x8bf63a to 0x5111b8 (0x3ae481 bytes) 000275[0] 1790786156.605086 read 0x38cf6a thru 0x38ee7f (0x1f16 bytes) 000276[0] 1790786156.651404 trunc from 0x5111b9 to 0x1ca16c (0x34704c bytes) 000279[0] 1790786156.746679 write 0x73b85e thru 0x73d357 (0x1afa bytes) 000280[1] 1790786156.753123 mapwrite 0x0b8d6e thru 0x0b97f6 (0xa89 bytes) 000281[1] 1790786156.912211 read 0x3bd571 thru 0x3bf3a1 (0x1e31 bytes) 000282[1] 1790786156.955143 mapread 0x42f3ba thru 0x43193d (0x2584 bytes) 000285[1] 1790786157.023971 mapwrite 0x209ec6 thru 0x20bb40 (0x1c7b bytes) 000286[1] 1790786157.120443 punch from 0x4f35cf to 0x4f525d (0x1c8f bytes) 000289[1] 1790786157.191105 trunc from 0x73d358 to 0x41e24a (0x31f10d bytes) 000290[1] 1790786157.229037 mapread 0x2f1b76 thru 0x2f7798 (0x5c23 bytes) 000293[0] 1790786157.248301 mapread 0x184e9a thru 0x1856d8 (0x83f bytes) truncating to largest ever: 0x9f4e8e 000295[1] 1790786157.311328 trunc from 0x41e24b to 0x9f4e8d (0x5d6c43 bytes) 000297[0] 1790786157.364027 punch from 0x5e12a4 to 0x5eba54 (0xa7b1 bytes) 000298[0] 1790786157.388030 mapwrite 0x0f8405 thru 0x0f8481 (0x7d bytes) 000299[1] 1790786157.542828 mapread 0x2cd809 thru 0x2d6556 (0x8d4e bytes) 000304[0] 1790786157.636903 read 0x5fc97e thru 0x6088f4 (0xbf77 bytes) 000307[0] 1790786157.692191 read 0x8025b3 thru 0x807057 (0x4aa5 bytes) 000308[0] 1790786157.712868 punch from 0x92c5f7 to 0x935598 (0x8fa2 bytes) 000311[1] 1790786157.763757 read 0x485bc6 thru 0x494401 (0xe83c bytes) 000314[0] 1790786157.902574 write 0x3da5c2 thru 0x3db977 (0x13b6 bytes) 000316[1] 1790786157.923447 punch from 0x22b85f to 0x22ed20 (0x34c2 bytes) 000317[0] 1790786158.004886 read 0x43b10b thru 0x440d67 (0x5c5d bytes) 000319[0] 1790786158.061504 read 0x9b053f thru 0x9bf340 (0xee02 bytes) 000320[1] 1790786158.098559 read 0x1f83f6 thru 0x2023c5 (0x9fd0 bytes) 000322[0] 1790786158.117669 punch from 0x6f1081 to 0x700523 (0xf4a3 bytes) 000324[1] 1790786158.169965 write 0x277beb thru 0x283223 (0xb639 bytes) 000327[0] 1790786158.242309 trunc from 0x9f4e8e to 0x1102ac (0x8e4be1 bytes) 000328[0] 1790786158.356114 write 0x427d1b thru 0x4328a8 (0xab8e bytes) 000329[0] 1790786158.363166 punch from 0x1511d7 to 0x151ea0 (0xcca bytes) 000332[1] 1790786158.439786 mapread 0x3416a7 thru 0x34a40c (0x8d66 bytes) 000334[0] 1790786158.684141 mapwrite 0x042407 thru 0x044807 (0x2401 bytes) 000336[1] 1790786158.896891 punch from 0x37ee0e to 0x384b53 (0x5d46 bytes) 000337[0] 1790786158.940706 mapwrite 0x522154 thru 0x52c811 (0xa6be bytes) 000338[0] 1790786159.197117 mapread 0x2f1ab7 thru 0x301915 (0xfe5f bytes) 000340[0] 1790786159.248886 trunc from 0x52c812 to 0x78b404 (0x25ebf3 bytes) 000341[0] 1790786159.370432 mapread 0x4aa5b8 thru 0x4ba245 (0xfc8e bytes) 000342[1] 1790786159.520403 punch from 0x543659 to 0x54a972 (0x731a bytes) 000343[0] 1790786159.598463 trunc from 0x78b405 to 0x362be9 (0x42881b bytes) 000344[0] 1790786159.681613 write 0x3c74af thru 0x3d243b (0xaf8d bytes) 000345[0] 1790786159.716029 trunc from 0x3d243c to 0x8d720d (0x504dd2 bytes) 000347[0] 1790786159.835447 write 0x4f9e96 thru 0x4fc6ae (0x2819 bytes) 000348[1] 1790786159.864051 trunc from 0x8d720e to 0x23db3b (0x6996d2 bytes) 000349[0] 1790786159.975973 mapwrite 0x5537ec thru 0x557f84 (0x4799 bytes) 000350[0] 1790786160.131400 punch from 0x328250 to 0x333521 (0xb2d2 bytes) 000353[1] 1790786160.178736 mapread 0x409c80 thru 0x40b15b (0x14dc bytes) 000357[1] 1790786160.238228 mapread 0x4cefef thru 0x4cfbca (0xbdc bytes) 000358[0] 1790786160.273834 mapwrite 0x01fe52 thru 0x027c4a (0x7df9 bytes) 000361[0] 1790786160.529059 mapwrite 0x3f5022 thru 0x3f66c0 (0x169f bytes) 000362[1] 1790786160.627895 trunc from 0x557f85 to 0x42a57e (0x12da06 bytes) 000364[1] 1790786160.735742 read 0x2513fa thru 0x2514ab (0xb2 bytes) 000367[1] 1790786160.834414 mapread 0x03382c thru 0x034892 (0x1067 bytes) 000368[1] 1790786160.895809 mapwrite 0x1a86aa thru 0x1b15ff (0x8f56 bytes) 000370[1] 1790786161.132048 mapwrite 0x173691 thru 0x18151a (0xde8a bytes) 000371[1] 1790786161.254125 punch from 0x35ce62 to 0x35ea30 (0x1bcf bytes) 000373[1] 1790786161.338388 trunc from 0x42a57f to 0x75cf6a (0x3329ec bytes) 000375[1] 1790786161.470371 trunc from 0x75cf6b to 0x6b5d4b (0xa721f bytes) 000377[0] 1790786161.606942 read 0x24b4c9 thru 0x24b752 (0x28a bytes) 000379[0] 1790786161.667626 mapread 0x5c1991 thru 0x5c7f82 (0x65f2 bytes) 000380[1] 1790786161.722848 read 0x58ba57 thru 0x594d80 (0x932a bytes) 000382[1] 1790786161.761003 mapwrite 0x023435 thru 0x029c1e (0x67ea bytes) 000383[0] 1790786162.027061 mapread 0x03fbe2 thru 0x0495f6 (0x9a15 bytes) 000385[0] 1790786162.103641 trunc from 0x6b5d4c to 0x3e46ea (0x2d1661 bytes) 000388[0] 1790786162.207457 mapread 0x029e62 thru 0x037e8e (0xe02d bytes) 000391[1] 1790786162.303555 trunc from 0x3e46eb to 0x92c5dd (0x547ef3 bytes) 000392[0] 1790786162.378675 read 0x39a6b6 thru 0x3a3613 (0x8f5e bytes) 000393[0] 1790786162.417643 read 0x398f70 thru 0x39c5ad (0x363e bytes) 000397[0] 1790786162.462393 trunc from 0x92c5de to 0x65275e (0x2d9e7f bytes) 000398[1] 1790786162.506836 mapread 0x0d238b thru 0x0d4689 (0x22ff bytes) 000399[1] 1790786162.558731 read 0x2ea818 thru 0x2f2ac1 (0x82aa bytes) 000400[1] 1790786162.574108 mapread 0x502bed thru 0x5031f1 (0x605 bytes) 000401[1] 1790786162.598965 mapwrite 0x305dc7 thru 0x309636 (0x3870 bytes) 000405[0] 1790786162.790850 mapwrite 0x11e4ce thru 0x12c4be (0xdff1 bytes) 000406[1] 1790786162.949667 mapwrite 0x8b1a8e thru 0x8b4617 (0x2b8a bytes) 000407[1] 1790786163.251768 mapread 0x642f1a thru 0x64ea86 (0xbb6d bytes) 000408[1] 1790786163.316902 mapread 0x3b2d2f thru 0x3c2a74 (0xfd46 bytes) 000409[1] 1790786163.398111 write 0x0ac20e thru 0x0b4791 (0x8584 bytes) 000410[0] 1790786163.416381 mapread 0x6766f9 thru 0x67e100 (0x7a08 bytes) 000411[0] 1790786163.531398 trunc from 0x8b4618 to 0x006a00 (0x8adc17 bytes) 000412[0] 1790786163.668736 mapread 0x0026d1 thru 0x006a00 (0x4330 bytes) 000414[1] 1790786163.693921 mapwrite 0x09d0ae thru 0x0ac0ea (0xf03d bytes) 000415[1] 1790786164.028209 write 0x58203a thru 0x5879b6 (0x597d bytes) 000418[1] 1790786164.036892 trunc from 0x5879b7 to 0x9828af (0x3faef9 bytes) 000419[0] 1790786164.099098 trunc from 0x9828b0 to 0x582952 (0x3fff5d bytes) 000420[1] 1790786164.162609 trunc from 0x582953 to 0x86c123 (0x2e97d1 bytes) 000423[0] 1790786164.274710 trunc from 0x86c124 to 0x6da276 (0x191ead bytes) 000424[1] 1790786164.349556 mapwrite 0x0297a4 thru 0x02f05d (0x58ba bytes) 000426[0] 1790786164.533169 write 0x9e0929 thru 0x9ef1e9 (0xe8c1 bytes) 000427[0] 1790786164.583567 write 0x9e8955 thru 0x9f5cf8 (0xd3a4 bytes) 000428[0] 1790786164.609438 mapwrite 0x6d3151 thru 0x6ddda1 (0xac51 bytes) 000429[0] 1790786164.726977 trunc from 0x9f5cf9 to 0x4db4d7 (0x51a821 bytes) 000430[0] 1790786164.766218 mapread 0x1a2a8f thru 0x1a527f (0x27f1 bytes) 000433[0] 1790786164.822344 punch from 0x1d9ccd to 0x1e7d21 (0xe055 bytes) 000434[0] 1790786164.837743 mapwrite 0x4473a4 thru 0x44f0d1 (0x7d2e bytes) 000440[0] 1790786165.061936 trunc from 0x4db4d8 to 0x3efb76 (0xeb961 bytes) 000442[1] 1790786165.168020 trunc from 0x3efb77 to 0x5f97ae (0x209c38 bytes) 000444[1] 1790786165.263634 punch from 0x13481d to 0x138e6b (0x464f bytes) 000446[0] 1790786165.268249 mapread 0x1fd068 thru 0x204ba0 (0x7b39 bytes) 000447[0] 1790786165.360405 write 0x9a7d3d thru 0x9ad9b1 (0x5c75 bytes) 000450[1] 1790786165.495722 punch from 0x7c0804 to 0x7c19d9 (0x11d6 bytes) 000451[1] 1790786165.547840 punch from 0x68f80f to 0x697287 (0x7a79 bytes) 000454[1] 1790786165.653116 write 0x79ea82 thru 0x7a1f9d (0x351c bytes) 000455[0] 1790786165.673260 write 0x962fbd thru 0x96fe48 (0xce8c bytes) 000457[1] 1790786165.717005 mapwrite 0x3068f0 thru 0x30c753 (0x5e64 bytes) 000458[0] 1790786165.952593 write 0x353086 thru 0x35454b (0x14c6 bytes) 000462[0] 1790786166.052314 trunc from 0x9ad9b2 to 0x401a2d (0x5abf84 bytes) 000464[1] 1790786166.162372 mapread 0x387296 thru 0x3943b5 (0xd120 bytes) 000467[1] 1790786166.318900 read 0x3a60aa thru 0x3b0c53 (0xabaa bytes) 000469[1] 1790786166.353520 mapread 0x2334b5 thru 0x2414d7 (0xe023 bytes) 000470[0] 1790786166.427093 write 0x527742 thru 0x528de3 (0x16a2 bytes) 000471[0] 1790786166.428313 trunc from 0x528de4 to 0x24565a (0x2e3789 bytes) 000473[1] 1790786166.518821 trunc from 0x24565b to 0x574e60 (0x32f806 bytes) 000474[0] 1790786166.660054 mapread 0x097354 thru 0x09aa9f (0x374c bytes) 000475[1] 1790786166.733910 punch from 0x06085d to 0x069b25 (0x92c9 bytes) 000477[0] 1790786166.864595 mapwrite 0x01b91d thru 0x01d3ee (0x1ad2 bytes) 000479[1] 1790786167.071778 trunc from 0x574e61 to 0x441609 (0x133857 bytes) 000480[0] 1790786167.156215 trunc from 0x44160a to 0x5052c6 (0xc3cbd bytes) 000482[0] 1790786167.313576 mapread 0x2df036 thru 0x2e33c3 (0x438e bytes) 000483[0] 1790786167.407778 mapread 0x3cbe21 thru 0x3cf45c (0x363c bytes) 000484[0] 1790786167.454554 mapwrite 0x0cfa6b thru 0x0db302 (0xb898 bytes) 000486[1] 1790786167.743802 read 0x2424ce thru 0x24c1ce (0x9d01 bytes) 000488[0] 1790786167.815235 read 0x38d533 thru 0x396163 (0x8c31 bytes) 000489[0] 1790786167.862995 read 0x213ec4 thru 0x219817 (0x5954 bytes) 000491[0] 1790786167.917454 mapread 0x36aa79 thru 0x370ae9 (0x6071 bytes) 000494[0] 1790786167.976366 write 0x5b0094 thru 0x5b4bf5 (0x4b62 bytes) 000495[0] 1790786167.992892 write 0x2080ed thru 0x217c6f (0xfb83 bytes) 000496[0] 1790786168.041643 mapread 0x37256d thru 0x37e60c (0xc0a0 bytes) 000497[0] 1790786168.066035 mapwrite 0x3f64f1 thru 0x3fe6f8 (0x8208 bytes) 000500[0] 1790786168.201291 write 0x3c48b3 thru 0x3cc169 (0x78b7 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6168 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1790786169.053603 write 0x2c4000 thru 0x2cd900 (0x9901 bytes) 000005[0] 1790786169.141629 mapwrite 0x638000 thru 0x64503f (0xd040 bytes) 000006[1] 1790786169.367162 mapwrite 0x7a1000 thru 0x7a4688 (0x3689 bytes) 000007[0] 1790786169.565283 mapread 0x5d1000 thru 0x5e099c (0xf99d bytes) 000008[0] 1790786169.661231 write_OD 0x82b000 thru 0x839fff (0xf000 bytes) 000009[1] 1790786169.772949 write 0x73b000 thru 0x742528 (0x7529 bytes) 000010[0] 1790786169.805668 read_OD 0x1d7000 thru 0x1e2fff (0xc000 bytes) 000011[0] 1790786169.842585 read_OD 0x258000 thru 0x261fff (0xa000 bytes) 000014[1] 1790786169.853289 read 0x304000 thru 0x30ea9e (0xaa9f bytes) 000015[0] 1790786169.864068 write_OD 0x2fa000 thru 0x2fafff (0x1000 bytes) 000016[0] 1790786169.914304 write_OD 0x0c7000 thru 0x0d4fff (0xe000 bytes) 000019[0] 1790786170.026211 write_OD 0x4d5000 thru 0x4dbfff (0x7000 bytes) 000020[0] 1790786170.091670 mapread 0x126000 thru 0x12e459 (0x845a bytes) 000022[1] 1790786170.127672 read 0x19a000 thru 0x19f457 (0x5458 bytes) 000023[1] 1790786170.135334 write 0x497000 thru 0x4a1ce6 (0xace7 bytes) 000024[1] 1790786170.167042 mapread 0x472000 thru 0x480c34 (0xec35 bytes) 000025[1] 1790786170.217807 read 0x2e9000 thru 0x2f1233 (0x8234 bytes) 000026[1] 1790786170.228157 write 0x27c000 thru 0x28729c (0xb29d bytes) 000027[1] 1790786170.246770 mapwrite 0x0a3000 thru 0x0b2cb8 (0xfcb9 bytes) 000028[1] 1790786170.358377 read 0x3ab000 thru 0x3ad620 (0x2621 bytes) truncating to largest ever: 0x22e327 000029[1] 1790786170.384056 trunc from 0x83a000 to 0x22e326 (0x60bcd9 bytes) 000030[0] 1790786170.409586 mapread 0x167000 thru 0x169b60 (0x2b61 bytes) 000031[0] 1790786170.450591 read_OD 0x1c8000 thru 0x1c9fff (0x2000 bytes) 000032[1] 1790786170.457354 mapread 0x08a000 thru 0x08ccec (0x2ced bytes) 000034[0] 1790786170.482401 mapread 0x22c000 thru 0x22d50a (0x150b bytes) 000037[1] 1790786170.526896 write 0x631000 thru 0x63e2e6 (0xd2e7 bytes) 000039[1] 1790786170.552597 mapread 0x0e8000 thru 0x0f1819 (0x981a bytes) 000040[0] 1790786170.621930 read_OD 0x189000 thru 0x18dfff (0x5000 bytes) 000041[0] 1790786170.658997 read_OD 0x092000 thru 0x09dfff (0xc000 bytes) 000044[0] 1790786170.673077 mapwrite 0x853000 thru 0x85d89e (0xa89f bytes) 000046[1] 1790786170.793761 mapread 0x27e000 thru 0x28c543 (0xe544 bytes) 000050[1] 1790786170.864743 mapwrite 0x8cf000 thru 0x8d75d9 (0x85da bytes) 000051[1] 1790786171.056367 mapwrite 0x326000 thru 0x32ba63 (0x5a64 bytes) 000052[1] 1790786171.184735 mapread 0x43e000 thru 0x4499bf (0xb9c0 bytes) 000053[0] 1790786171.218052 punch from 0x17a9db to 0x18a5bc (0xfbe2 bytes) 000054[1] 1790786171.230238 mapwrite 0x1d9000 thru 0x1dd7f6 (0x47f7 bytes) 000056[1] 1790786171.319681 write 0x45a000 thru 0x45e4d7 (0x44d8 bytes) 000057[0] 1790786171.323956 mapread 0x702000 thru 0x705db0 (0x3db1 bytes) 000059[1] 1790786171.377707 write 0x550000 thru 0x554b7f (0x4b80 bytes) 000060[0] 1790786171.404171 punch from 0x531f4b to 0x54111b (0xf1d1 bytes) 000062[1] 1790786171.463034 read 0x188000 thru 0x18ae51 (0x2e52 bytes) 000064[1] 1790786171.505399 write 0x56d000 thru 0x57cb6f (0xfb70 bytes) 000065[0] 1790786171.564221 mapwrite 0x4ca000 thru 0x4d157c (0x757d bytes) 000067[0] 1790786171.747615 trunc from 0x8d75da to 0x1c2c99 (0x714940 bytes) 000068[1] 1790786171.771389 punch from 0x1b2b5d to 0x1bd69d (0xab41 bytes) 000070[0] 1790786171.829407 mapwrite 0x57e000 thru 0x5840cd (0x60ce bytes) truncating to largest ever: 0x5778dc 000071[1] 1790786171.929539 trunc from 0x5840ce to 0x5778db (0xc7f2 bytes) 000072[0] 1790786171.959200 read_OD 0x473000 thru 0x47ffff (0xd000 bytes) 000075[1] 1790786172.001796 mapwrite 0x99c000 thru 0x9a985a (0xd85b bytes) 000076[0] 1790786172.248402 mapwrite 0x642000 thru 0x648027 (0x6028 bytes) 000079[1] 1790786172.487402 punch from 0x3e26ac to 0x3ea029 (0x797e bytes) 000083[0] 1790786172.556263 read_OD 0x682000 thru 0x683fff (0x2000 bytes) 000084[0] 1790786172.591438 read_OD 0x942000 thru 0x94bfff (0xa000 bytes) 000085[1] 1790786172.614979 mapwrite 0x57f000 thru 0x583532 (0x4533 bytes) 000089[1] 1790786172.791788 punch from 0x58a79e to 0x59a764 (0xffc7 bytes) 000090[0] 1790786172.816095 mapread 0x099000 thru 0x0a1832 (0x8833 bytes) 000091[1] 1790786172.864172 read 0x65d000 thru 0x65e553 (0x1554 bytes) 000093[1] 1790786172.871933 write 0x1dc000 thru 0x1dd32f (0x1330 bytes) truncating to largest ever: 0x98bef7 000096[1] 1790786172.920067 trunc from 0x9a985b to 0x98bef6 (0x1d964 bytes) 000097[0] 1790786172.978224 trunc from 0x98bef7 to 0x853685 (0x138871 bytes) 000098[0] 1790786173.050715 trunc from 0x853686 to 0x648ecb (0x20a7ba bytes) 000102[0] 1790786173.125344 mapwrite 0x827000 thru 0x82f8d0 (0x88d1 bytes) 000103[1] 1790786173.233167 mapwrite 0x36f000 thru 0x3714c7 (0x24c8 bytes) 000111[1] 1790786173.383599 write 0x8c3000 thru 0x8cb70a (0x870b bytes) 000113[0] 1790786173.385118 mapwrite 0x12f000 thru 0x13b139 (0xc13a bytes) 000114[1] 1790786173.551769 mapwrite 0x363000 thru 0x36de9e (0xae9f bytes) 000118[1] 1790786173.703554 punch from 0x758025 to 0x75f882 (0x785e bytes) 000119[0] 1790786173.726736 read_OD 0x52f000 thru 0x536fff (0x8000 bytes) 000120[0] 1790786173.772703 write_OD 0x7a7000 thru 0x7b4fff (0xe000 bytes) 000121[1] 1790786173.887608 trunc from 0x8cb70b to 0x7bbe12 (0x10f8f8 bytes) 000123[1] 1790786173.976752 write 0x786000 thru 0x78b326 (0x5327 bytes) 000125[0] 1790786174.006537 mapwrite 0x773000 thru 0x77e034 (0xb035 bytes) 000126[1] 1790786174.145612 mapwrite 0x08c000 thru 0x099dec (0xdded bytes) 000128[1] 1790786174.370656 read 0x0ef000 thru 0x0fe3dc (0xf3dd bytes) 000130[1] 1790786174.388918 write 0x482000 thru 0x48b748 (0x9749 bytes) 000132[0] 1790786174.420718 trunc from 0x7bbe13 to 0x28f622 (0x52c7f0 bytes) 000133[1] 1790786174.484442 write 0x435000 thru 0x4402f2 (0xb2f3 bytes) 000135[1] 1790786174.499592 punch from 0x34ea90 to 0x35e71c (0xfc8d bytes) 000136[0] 1790786174.519288 read_OD 0x0e3000 thru 0x0e7fff (0x5000 bytes) 000137[1] 1790786174.548781 mapwrite 0x73c000 thru 0x7432cd (0x72ce bytes) 000140[1] 1790786174.754586 mapwrite 0x4b1000 thru 0x4bebc9 (0xdbca bytes) 000142[0] 1790786174.979582 punch from 0x4dfaa6 to 0x4ef4d9 (0xfa34 bytes) 000144[1] 1790786175.012534 trunc from 0x7432ce to 0x0a9f40 (0x69938d bytes) 000145[1] 1790786175.098424 write 0x145000 thru 0x147358 (0x2359 bytes) 000146[0] 1790786175.138881 read_OD 0x037000 thru 0x045fff (0xf000 bytes) 000149[0] 1790786175.198333 read_OD 0x07b000 thru 0x080fff (0x6000 bytes) 000150[0] 1790786175.223684 punch from 0x041773 to 0x0443ea (0x2c78 bytes) 000151[0] 1790786175.247850 mapread 0x0cc000 thru 0x0d6a6b (0xaa6c bytes) 000152[0] 1790786175.305345 mapwrite 0x020000 thru 0x02d889 (0xd88a bytes) 000154[1] 1790786175.363018 write 0x309000 thru 0x316f13 (0xdf14 bytes) 000158[0] 1790786175.415498 punch from 0x007a9a to 0x00aef1 (0x3458 bytes) 000160[1] 1790786175.478774 mapread 0x001000 thru 0x0100f6 (0xf0f7 bytes) 000164[0] 1790786175.528693 punch from 0x0e0316 to 0x0e8fe5 (0x8cd0 bytes) 000165[0] 1790786175.548336 write_OD 0x092000 thru 0x0a0fff (0xf000 bytes) 000166[1] 1790786175.597455 trunc from 0x316f14 to 0x4090a5 (0xf2192 bytes) 000168[1] 1790786175.625414 trunc from 0x4090a6 to 0x5dec8e (0x1d5be9 bytes) 000169[1] 1790786175.653982 punch from 0x12cfce to 0x12ff3d (0x2f70 bytes) 000170[1] 1790786175.671727 read 0x5a6000 thru 0x5a87be (0x27bf bytes) 000172[1] 1790786175.710985 mapread 0x20f000 thru 0x212001 (0x3002 bytes) 000176[0] 1790786175.798759 mapwrite 0x1b3000 thru 0x1bdac3 (0xaac4 bytes) 000177[1] 1790786175.976975 mapwrite 0x1ac000 thru 0x1b99af (0xd9b0 bytes) 000179[1] 1790786176.196908 read 0x24f000 thru 0x2523f1 (0x33f2 bytes) 000181[1] 1790786176.222941 write 0x58d000 thru 0x58faa5 (0x2aa6 bytes) 000185[0] 1790786176.265634 read_OD 0x1c4000 thru 0x1cffff (0xc000 bytes) 000187[1] 1790786176.349190 mapread 0x342000 thru 0x35111b (0xf11c bytes) 000188[1] 1790786176.415212 mapread 0x48b000 thru 0x492bf0 (0x7bf1 bytes) 000189[0] 1790786176.462172 punch from 0x1622e9 to 0x17172e (0xf446 bytes) 000190[0] 1790786176.486728 trunc from 0x5dec8f to 0x316719 (0x2c8575 bytes) 000191[1] 1790786176.540131 mapwrite 0x1fc000 thru 0x1fd9bf (0x19c0 bytes) 000193[0] 1790786176.652302 mapwrite 0x357000 thru 0x35a580 (0x3581 bytes) 000194[1] 1790786176.781169 mapwrite 0x635000 thru 0x63d187 (0x8188 bytes) 000195[1] 1790786176.957592 read 0x26a000 thru 0x2741f3 (0xa1f4 bytes) 000196[1] 1790786176.990403 mapread 0x3bb000 thru 0x3c7b2a (0xcb2b bytes) 000197[1] 1790786177.019267 punch from 0x2a3729 to 0x2ac928 (0x9200 bytes) 000198[1] 1790786177.043763 mapread 0x45d000 thru 0x464d7b (0x7d7c bytes) 000199[1] 1790786177.081964 mapwrite 0x6ad000 thru 0x6b57dc (0x87dd bytes) 000200[0] 1790786177.205648 write_OD 0x81a000 thru 0x822fff (0x9000 bytes) 000201[1] 1790786177.305393 trunc from 0x823000 to 0x9844dc (0x1614dd bytes) 000202[0] 1790786177.330875 trunc from 0x9844dd to 0x2298b3 (0x75ac29 bytes) 000203[1] 1790786177.386831 write 0x75c000 thru 0x767829 (0xb82a bytes) 000204[0] 1790786177.439349 mapwrite 0x32a000 thru 0x337ffe (0xdfff bytes) 000205[1] 1790786177.578021 mapwrite 0x674000 thru 0x67b879 (0x787a bytes) 000207[1] 1790786177.720659 write 0x429000 thru 0x43192d (0x892e bytes) 000209[1] 1790786177.731400 read 0x314000 thru 0x317c89 (0x3c8a bytes) truncating to largest ever: 0x9d6a4d 000211[0] 1790786177.765890 trunc from 0x76782a to 0x9d6a4c (0x26f223 bytes) 000213[1] 1790786177.863300 punch from 0x557c8a to 0x55c371 (0x46e8 bytes) 000214[1] 1790786177.877692 mapread 0x6e0000 thru 0x6e190a (0x190b bytes) 000215[1] 1790786177.897784 punch from 0x41fbcd to 0x420593 (0x9c7 bytes) 000216[0] 1790786177.924716 mapwrite 0x701000 thru 0x70a88a (0x988b bytes) 000218[0] 1790786178.101853 punch from 0x2fd2a1 to 0x300022 (0x2d82 bytes) 000219[0] 1790786178.112150 punch from 0x3d56ab to 0x3d662d (0xf83 bytes) 000220[0] 1790786178.131750 punch from 0x3e6ead to 0x3f63ef (0xf543 bytes) 000221[0] 1790786178.174430 punch from 0x5f9d54 to 0x5fcf19 (0x31c6 bytes) 000222[1] 1790786178.181317 trunc from 0x9d6a4d to 0x8a3b51 (0x132efb bytes) 000224[1] 1790786178.249010 write 0x34e000 thru 0x351557 (0x3558 bytes) 000227[1] 1790786178.269721 write 0x8b4000 thru 0x8c2d52 (0xed53 bytes) 000228[0] 1790786178.284342 mapread 0x13d000 thru 0x14bfa0 (0xefa1 bytes) 000229[0] 1790786178.342622 trunc from 0x8c2d53 to 0x57f60a (0x343748 bytes) 000230[1] 1790786178.387644 write 0x637000 thru 0x63c903 (0x5904 bytes) 000232[0] 1790786178.415369 mapwrite 0x4c0000 thru 0x4c24c5 (0x24c6 bytes) 000233[0] 1790786178.522208 punch from 0x403fd0 to 0x40d7ec (0x981d bytes) 000234[0] 1790786178.544307 read_OD 0x12f000 thru 0x13bfff (0xd000 bytes) 000235[0] 1790786178.566453 punch from 0x12aa75 to 0x135d41 (0xb2cd bytes) 000236[0] 1790786178.618330 write_OD 0x627000 thru 0x628fff (0x2000 bytes) 000238[1] 1790786178.685143 write 0x6c9000 thru 0x6d59da (0xc9db bytes) 000239[0] 1790786178.697608 mapwrite 0x889000 thru 0x893de2 (0xade3 bytes) 000240[1] 1790786178.840434 write 0x5df000 thru 0x5e22e1 (0x32e2 bytes) 000241[1] 1790786178.886919 write 0x045000 thru 0x04c6f1 (0x76f2 bytes) 000243[0] 1790786178.914370 read_OD 0x850000 thru 0x853fff (0x4000 bytes) 000244[1] 1790786178.971093 punch from 0x447afc to 0x456e32 (0xf337 bytes) 000245[1] 1790786179.020793 punch from 0x6c85db to 0x6caac2 (0x24e8 bytes) 000247[1] 1790786179.030138 trunc from 0x893de3 to 0x451868 (0x44257a bytes) 000248[1] 1790786179.069224 write 0x6df000 thru 0x6eb39f (0xc3a0 bytes) 000249[1] 1790786179.102870 mapwrite 0x369000 thru 0x3760e3 (0xd0e4 bytes) 000250[1] 1790786179.304351 read 0x230000 thru 0x23b077 (0xb078 bytes) 000251[0] 1790786179.327847 write_OD 0x23f000 thru 0x247fff (0x9000 bytes) 000252[1] 1790786179.386783 punch from 0x201c73 to 0x20c681 (0xaa0f bytes) 000253[0] 1790786179.414756 trunc from 0x6eb3a0 to 0x3c1360 (0x32a03f bytes) 000254[0] 1790786179.453667 mapread 0x27f000 thru 0x280ba2 (0x1ba3 bytes) 000255[1] 1790786179.477667 write 0x4a6000 thru 0x4add4b (0x7d4c bytes) 000256[1] 1790786179.506706 punch from 0x2f52ee to 0x303671 (0xe384 bytes) 000257[0] 1790786179.511996 read_OD 0x3e7000 thru 0x3f3fff (0xd000 bytes) 000258[0] 1790786179.544249 read_OD 0x27c000 thru 0x27efff (0x3000 bytes) 000259[1] 1790786179.554581 mapread 0x43f000 thru 0x43fb3d (0xb3e bytes) 000260[0] 1790786179.561791 mapread 0x153000 thru 0x16104a (0xe04b bytes) 000261[1] 1790786179.632314 punch from 0x185394 to 0x185659 (0x2c6 bytes) 000262[1] 1790786179.690320 mapwrite 0x6a3000 thru 0x6a5b06 (0x2b07 bytes) 000263[0] 1790786179.933087 punch from 0x53a5f5 to 0x53b43b (0xe47 bytes) 000266[0] 1790786180.037257 read_OD 0x110000 thru 0x114fff (0x5000 bytes) 000271[1] 1790786180.082678 mapwrite 0x215000 thru 0x218dda (0x3ddb bytes) 000272[0] 1790786180.267396 mapread 0x26d000 thru 0x276865 (0x9866 bytes) 000274[0] 1790786180.358549 write_OD 0x2d7000 thru 0x2d8fff (0x2000 bytes) 000275[1] 1790786180.450082 mapwrite 0x937000 thru 0x941879 (0xa87a bytes) 000278[1] 1790786180.693279 punch from 0x2bd819 to 0x2cc1e5 (0xe9cd bytes) 000279[0] 1790786180.721945 write_OD 0x07d000 thru 0x07dfff (0x1000 bytes) 000280[1] 1790786180.814696 write 0x77d000 thru 0x78cb4c (0xfb4d bytes) 000281[1] 1790786180.876824 mapwrite 0x923000 thru 0x930233 (0xd234 bytes) 000282[1] 1790786181.052100 mapread 0x283000 thru 0x283155 (0x156 bytes) 000283[0] 1790786181.092740 punch from 0x932ef1 to 0x9407e6 (0xd8f6 bytes) 000289[1] 1790786181.260723 mapread 0x865000 thru 0x86fbc7 (0xabc8 bytes) 000292[1] 1790786181.358013 mapwrite 0x7b1000 thru 0x7be8a7 (0xd8a8 bytes) 000293[0] 1790786181.599009 punch from 0x348ce3 to 0x34fd87 (0x70a5 bytes) 000294[0] 1790786181.653885 trunc from 0x94187a to 0x2af1b1 (0x6926c8 bytes) 000298[0] 1790786181.726812 read_OD 0x230000 thru 0x237fff (0x8000 bytes) 000299[1] 1790786181.764089 mapwrite 0x03f000 thru 0x04198a (0x298b bytes) 000303[0] 1790786181.879369 punch from 0x015579 to 0x0174a0 (0x1f28 bytes) 000304[1] 1790786181.919699 write 0x2bc000 thru 0x2c7fe0 (0xbfe1 bytes) 000306[0] 1790786181.975331 mapread 0x108000 thru 0x10f805 (0x7806 bytes) 000307[0] 1790786182.095684 write_OD 0x436000 thru 0x43cfff (0x7000 bytes) 000308[0] 1790786182.203769 trunc from 0x43d000 to 0x5e9064 (0x1ac065 bytes) 000311[0] 1790786182.318559 mapread 0x0ae000 thru 0x0b2b6d (0x4b6e bytes) 000312[0] 1790786182.423414 mapwrite 0x992000 thru 0x99fa22 (0xda23 bytes) 000314[0] 1790786182.677134 trunc from 0x99fa23 to 0x4ac208 (0x4f381a bytes) 000316[0] 1790786182.725051 write_OD 0x5d6000 thru 0x5dcfff (0x7000 bytes) 000318[1] 1790786182.786773 mapwrite 0x657000 thru 0x65ed86 (0x7d87 bytes) 000319[0] 1790786182.923946 mapread 0x4e8000 thru 0x4f0a6a (0x8a6b bytes) 000320[0] 1790786183.006090 read_OD 0x606000 thru 0x612fff (0xd000 bytes) 000325[0] 1790786183.025684 trunc from 0x65ed87 to 0x8720ed (0x213367 bytes) 000327[1] 1790786183.097772 punch from 0x7461f8 to 0x74dc84 (0x7a8d bytes) 000328[1] 1790786183.142918 write 0x9b8000 thru 0x9c6d0a (0xed0b bytes) 000330[1] 1790786183.179381 punch from 0x1f8a85 to 0x207156 (0xe6d2 bytes) 000331[1] 1790786183.208315 read 0x916000 thru 0x91d56d (0x756e bytes) 000334[0] 1790786183.304458 write_OD 0x072000 thru 0x080fff (0xf000 bytes) 000336[0] 1790786183.445733 write_OD 0x36c000 thru 0x36efff (0x3000 bytes) 000337[0] 1790786183.552876 trunc from 0x9c6d0b to 0x0456a2 (0x981668 bytes) 000338[1] 1790786183.615532 write 0x3ec000 thru 0x3ef5ae (0x35af bytes) 000339[0] 1790786183.689515 trunc from 0x3ef5af to 0x4c8838 (0xd928a bytes) 000341[0] 1790786183.845254 trunc from 0x4c8839 to 0x9387b6 (0x46ff7e bytes) 000342[0] 1790786183.889766 write_OD 0x337000 thru 0x338fff (0x2000 bytes) 000345[0] 1790786184.008709 mapwrite 0x60b000 thru 0x61576d (0xa76e bytes) 000346[1] 1790786184.211341 trunc from 0x9387b7 to 0x4dd873 (0x45af43 bytes) 000348[1] 1790786184.331966 punch from 0x1c3a18 to 0x1c7b4f (0x4138 bytes) 000350[0] 1790786184.345350 read_OD 0x32e000 thru 0x338fff (0xb000 bytes) 000354[0] 1790786184.409267 read_OD 0x254000 thru 0x261fff (0xe000 bytes) 000355[1] 1790786184.433639 punch from 0x0cb631 to 0x0d31b8 (0x7b88 bytes) 000356[1] 1790786184.458869 write 0x4a4000 thru 0x4a856c (0x456d bytes) 000357[1] 1790786184.481633 read 0x277000 thru 0x282042 (0xb043 bytes) 000359[1] 1790786184.510326 trunc from 0x4dd874 to 0x3670cd (0x1767a6 bytes) 000361[0] 1790786184.567064 write_OD 0x995000 thru 0x999fff (0x5000 bytes) 000362[1] 1790786184.680117 read 0x058000 thru 0x063e02 (0xbe03 bytes) 000363[1] 1790786184.799824 write 0x55c000 thru 0x567e86 (0xbe87 bytes) 000364[0] 1790786184.911891 mapread 0x815000 thru 0x821522 (0xc523 bytes) 000365[1] 1790786185.031349 punch from 0x752a66 to 0x75745c (0x49f7 bytes) 000367[0] 1790786185.101612 read_OD 0x93b000 thru 0x946fff (0xc000 bytes) 000369[1] 1790786185.145931 mapread 0x54b000 thru 0x5531fe (0x81ff bytes) 000370[1] 1790786185.237330 mapread 0x090000 thru 0x09207f (0x2080 bytes) 000372[1] 1790786185.264883 mapwrite 0x379000 thru 0x383ad5 (0xaad6 bytes) 000374[1] 1790786185.515832 mapread 0x1f3000 thru 0x1fca1b (0x9a1c bytes) 000377[0] 1790786185.660065 read_OD 0x87e000 thru 0x88cfff (0xf000 bytes) 000378[0] 1790786185.704117 mapread 0x6e4000 thru 0x6f3946 (0xf947 bytes) 000379[0] 1790786185.734384 read_OD 0x8f5000 thru 0x902fff (0xe000 bytes) 000380[0] 1790786185.751730 trunc from 0x99a000 to 0x711ce4 (0x28831b bytes) 000382[0] 1790786185.808661 read_OD 0x551000 thru 0x55efff (0xe000 bytes) 000383[0] 1790786185.853681 mapread 0x072000 thru 0x07e38d (0xc38e bytes) 000385[1] 1790786185.923740 read 0x5c0000 thru 0x5c85d2 (0x85d3 bytes) 000386[0] 1790786185.954808 write_OD 0x147000 thru 0x153fff (0xd000 bytes) 000390[1] 1790786186.092693 mapread 0x3aa000 thru 0x3afbac (0x5bad bytes) 000391[1] 1790786186.153377 read 0x552000 thru 0x554c59 (0x2c5a bytes) 000393[0] 1790786186.166073 read_OD 0x165000 thru 0x165fff (0x1000 bytes) 000397[0] 1790786186.188226 mapwrite 0x3a9000 thru 0x3ad92c (0x492d bytes) 000399[1] 1790786186.329931 read 0x23d000 thru 0x24a5aa (0xd5ab bytes) 000400[1] 1790786186.359284 mapread 0x44d000 thru 0x458f21 (0xbf22 bytes) 000401[1] 1790786186.387340 mapread 0x1ca000 thru 0x1d7a1e (0xda1f bytes) 000402[1] 1790786186.422836 write 0x4dc000 thru 0x4e3d76 (0x7d77 bytes) 000405[0] 1790786186.467617 mapwrite 0x254000 thru 0x25ceec (0x8eed bytes) 000406[0] 1790786186.592749 mapwrite 0x82e000 thru 0x831676 (0x3677 bytes) 000409[1] 1790786186.715351 read 0x226000 thru 0x233077 (0xd078 bytes) 000410[0] 1790786186.744034 mapwrite 0x128000 thru 0x12d40a (0x540b bytes) 000411[0] 1790786186.899000 trunc from 0x831677 to 0x6e8726 (0x148f50 bytes) 000412[1] 1790786186.918629 punch from 0x40799a to 0x4101ed (0x8854 bytes) 000414[1] 1790786186.968865 mapread 0x2a7000 thru 0x2a8dd5 (0x1dd6 bytes) 000415[0] 1790786186.989866 read_OD 0x30e000 thru 0x313fff (0x6000 bytes) 000416[0] 1790786187.025059 trunc from 0x6e8727 to 0x12a901 (0x5bde25 bytes) 000417[0] 1790786187.057283 punch from 0x0c890d to 0x0d15f8 (0x8cec bytes) 000419[1] 1790786187.106789 mapread 0x125000 thru 0x12a13e (0x513f bytes) 000421[1] 1790786187.177285 read 0x069000 thru 0x075301 (0xc302 bytes) 000422[0] 1790786187.199777 trunc from 0x12a902 to 0x4ed223 (0x3c2922 bytes) 000423[1] 1790786187.269156 trunc from 0x4ed224 to 0x70e890 (0x22166d bytes) 000425[1] 1790786187.293590 mapread 0x310000 thru 0x31e3bb (0xe3bc bytes) 000426[0] 1790786187.396393 mapread 0x6f4000 thru 0x702c62 (0xec63 bytes) 000430[1] 1790786187.475741 mapwrite 0x676000 thru 0x680f53 (0xaf54 bytes) 000431[1] 1790786187.588521 write 0x95f000 thru 0x96cf36 (0xdf37 bytes) 000432[1] 1790786187.592233 punch from 0x3db9b2 to 0x3dc805 (0xe54 bytes) 000434[1] 1790786187.612901 trunc from 0x96cf37 to 0x2023e1 (0x76ab55 bytes) 000435[0] 1790786187.659265 trunc from 0x2023e2 to 0x04096d (0x1c1a74 bytes) 000438[1] 1790786187.728338 read 0x028000 thru 0x02d11a (0x511b bytes) 000440[0] 1790786187.769227 mapwrite 0x793000 thru 0x79f299 (0xc29a bytes) 000441[0] 1790786187.943753 mapread 0x3c3000 thru 0x3cfd22 (0xcd23 bytes) 000442[1] 1790786187.987851 read 0x026000 thru 0x03126e (0xb26f bytes) 000444[1] 1790786188.023942 trunc from 0x79f29a to 0x00cf55 (0x792344 bytes) 000445[0] 1790786188.075234 trunc from 0x00cf56 to 0x277c70 (0x26ad1b bytes) 000448[0] 1790786188.133491 read_OD 0x077000 thru 0x07dfff (0x7000 bytes) 000449[1] 1790786188.168580 trunc from 0x277c71 to 0x16a356 (0x10d91a bytes) 000452[1] 1790786188.214242 write 0x254000 thru 0x257661 (0x3662 bytes) 000455[1] 1790786188.265721 mapwrite 0x1fd000 thru 0x2052a3 (0x82a4 bytes) 000456[0] 1790786188.387335 read_OD 0x14c000 thru 0x157fff (0xc000 bytes) 000458[0] 1790786188.419112 mapread 0x0ed000 thru 0x0f3e3d (0x6e3e bytes) 000460[1] 1790786188.438976 punch from 0x10a794 to 0x113479 (0x8ce6 bytes) 000461[0] 1790786188.457741 punch from 0x055909 to 0x059db6 (0x44ae bytes) 000463[1] 1790786188.492645 mapread 0x1d3000 thru 0x1d8195 (0x5196 bytes) 000465[1] 1790786188.534773 trunc from 0x257662 to 0x95f875 (0x708214 bytes) 000466[0] 1790786188.568353 punch from 0x6d67aa to 0x6dd133 (0x698a bytes) 000467[1] 1790786188.593177 punch from 0x473676 to 0x4828b9 (0xf244 bytes) 000471[0] 1790786188.644704 punch from 0x888d83 to 0x893de6 (0xb064 bytes) 000472[0] 1790786188.665579 write_OD 0x147000 thru 0x152fff (0xc000 bytes) 000473[1] 1790786188.730844 trunc from 0x95f876 to 0x32e9ea (0x630e8b bytes) 000476[1] 1790786188.798669 read 0x2ca000 thru 0x2d081f (0x6820 bytes) 000480[0] 1790786188.899329 mapread 0x23e000 thru 0x24dc34 (0xfc35 bytes) 000482[1] 1790786189.036330 write 0x750000 thru 0x75ecbb (0xecbc bytes) 000484[0] 1790786189.055024 trunc from 0x75ecbc to 0x35e6a9 (0x400612 bytes) 000485[0] 1790786189.095924 mapread 0x0b3000 thru 0x0bc52f (0x9530 bytes) 000486[1] 1790786189.218813 trunc from 0x35e6aa to 0x0a9d9d (0x2b490c bytes) 000487[1] 1790786189.294779 punch from 0x05d6cd to 0x069487 (0xbdbb bytes) 000488[0] 1790786189.324608 punch from 0x012245 to 0x018159 (0x5f15 bytes) 000490[0] 1790786189.427048 read_OD 0x00a000 thru 0x00ffff (0x6000 bytes) 000491[1] 1790786189.474284 mapwrite 0x25e000 thru 0x2628d3 (0x48d4 bytes) 000493[0] 1790786189.759310 write_OD 0x537000 thru 0x53ffff (0x9000 bytes) 000495[0] 1790786189.852944 trunc from 0x540000 to 0x8ffda2 (0x3bfda3 bytes) 000497[0] 1790786189.947116 mapread 0x267000 thru 0x274ed2 (0xded3 bytes) 000498[1] 1790786190.045153 punch from 0x392bd2 to 0x3989da (0x5e09 bytes) 000499[1] 1790786190.095905 mapwrite 0x764000 thru 0x76d7a2 (0x97a3 bytes) All operations completed A-OK! PASS 16a (61s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 12:36:36 (1790786196) keep default fallocate mode: 2 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.444047 s, 18.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.307253 s, 27.3 MB/s Chance of close/open is 1 in 50 Seed set to 6200 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000006[1] 1790786202.002692 write 0x08e230 thru 0x08e2fd (0xce bytes) 000007[1] 1790786202.041671 read 0x02d2d9 thru 0x02d9df (0x707 bytes) 000014[1] 1790786202.185435 mapwrite 0x6baa15 thru 0x6bb861 (0xe4d bytes) 000016[0] 1790786202.360951 write 0x6926c8 thru 0x693082 (0x9bb bytes) truncating to largest ever: 0x9d4cae 000017[0] 1790786202.416582 trunc from 0x6bb862 to 0x9d4cad (0x31944c bytes) 000018[1] 1790786202.523910 punch from 0x717dfe to 0x717eec (0xef bytes) 000020[1] 1790786202.600723 read 0x95c3e5 thru 0x95ca9b (0x6b7 bytes) 000025[0] 1790786202.692426 read 0x794d26 thru 0x79661a (0x18f5 bytes) 000026[0] 1790786202.736665 punch from 0x124ece to 0x126a54 (0x1b87 bytes) 000029[1] 1790786202.795601 mapread 0x1ca976 thru 0x1cb471 (0xafc bytes) 000031[1] 1790786202.844706 mapwrite 0x3ff494 thru 0x3fffdf (0xb4c bytes) 000032[0] 1790786202.961424 write 0x700771 thru 0x700f42 (0x7d2 bytes) 000033[1] 1790786203.028204 mapwrite 0x045f6b thru 0x04739d (0x1433 bytes) 000036[0] 1790786203.352849 trunc from 0x9d4cae to 0x76c5de (0x2686cf bytes) 000039[0] 1790786203.468087 mapwrite 0x2004c3 thru 0x20162c (0x116a bytes) 000040[0] 1790786203.520792 mapwrite 0x27397b thru 0x273a4e (0xd4 bytes) 000041[0] 1790786203.584096 write 0x0910df thru 0x091cf8 (0xc1a bytes) 000042[1] 1790786203.597681 mapread 0x55665e thru 0x55670d (0xb0 bytes) 000044[0] 1790786203.660819 punch from 0x606650 to 0x607ca0 (0x1651 bytes) 000046[1] 1790786203.712296 mapwrite 0x551faf thru 0x553632 (0x1684 bytes) 000048[1] 1790786203.845564 punch from 0x12354e to 0x1246a9 (0x115c bytes) 000049[0] 1790786203.907083 punch from 0x3582c9 to 0x359e96 (0x1bce bytes) 000051[0] 1790786203.982317 write 0x69239c thru 0x6941a6 (0x1e0b bytes) 000052[0] 1790786204.022792 trunc from 0x76c5df to 0x5e1e5a (0x18a784 bytes) 000053[1] 1790786204.074614 mapwrite 0x9486e0 thru 0x949662 (0xf83 bytes) 000055[0] 1790786204.187970 mapwrite 0x67b4b9 thru 0x67cab1 (0x15f9 bytes) 000057[1] 1790786204.323275 mapwrite 0x4ad3d4 thru 0x4ae4b3 (0x10e0 bytes) 000058[0] 1790786204.431625 mapwrite 0x0d9958 thru 0x0da163 (0x80c bytes) 000059[0] 1790786204.607306 punch from 0x5b1d0f to 0x5b3b46 (0x1e38 bytes) 000061[0] 1790786204.677634 write 0x107f79 thru 0x109220 (0x12a8 bytes) 000062[0] 1790786204.722802 trunc from 0x949663 to 0x391366 (0x5b82fc bytes) 000063[1] 1790786204.752222 trunc from 0x391367 to 0x0b25ac (0x2dedba bytes) 000064[1] 1790786204.840656 mapwrite 0x1bc438 thru 0x1bc52d (0xf6 bytes) 000065[1] 1790786204.913212 mapwrite 0x7c2232 thru 0x7c2b62 (0x931 bytes) 000067[1] 1790786204.968964 write 0x8c9b5f thru 0x8ca61c (0xabe bytes) 000069[0] 1790786205.012749 write 0x8c7c86 thru 0x8c98f2 (0x1c6d bytes) 000072[0] 1790786205.123808 write 0x097709 thru 0x0987f1 (0x10e9 bytes) 000073[0] 1790786205.175752 write 0x4912cb thru 0x4926d0 (0x1406 bytes) 000077[1] 1790786205.286317 trunc from 0x8ca61d to 0x2dafa7 (0x5ef675 bytes) 000078[1] 1790786205.399498 write 0x7db23c thru 0x7dd12a (0x1eef bytes) 000079[1] 1790786205.430700 mapread 0x259478 thru 0x25a076 (0xbff bytes) 000080[0] 1790786205.462781 mapwrite 0x188c85 thru 0x18a563 (0x18df bytes) 000081[0] 1790786205.538360 mapread 0x3a446c thru 0x3a61bf (0x1d54 bytes) 000082[1] 1790786205.599800 mapwrite 0x2df99b thru 0x2e0703 (0xd69 bytes) 000083[0] 1790786205.673196 punch from 0x646a07 to 0x647772 (0xd6c bytes) 000084[0] 1790786205.731442 mapwrite 0x181b36 thru 0x1835bb (0x1a86 bytes) 000085[1] 1790786205.915949 read 0x3fc997 thru 0x3fe375 (0x19df bytes) 000087[1] 1790786205.987616 mapread 0x0f5474 thru 0x0f67aa (0x1337 bytes) 000088[1] 1790786206.045928 punch from 0x53b50a to 0x53c67a (0x1171 bytes) 000091[0] 1790786206.094959 mapread 0x0b20ab thru 0x0b2ce8 (0xc3e bytes) 000095[0] 1790786206.184531 trunc from 0x7dd12b to 0x843fc0 (0x66e96 bytes) 000096[0] 1790786206.256944 read 0x05d96a thru 0x05eeaf (0x1546 bytes) 000099[1] 1790786206.290857 trunc from 0x843fc1 to 0x8525bd (0xe5fd bytes) 000100[0] 1790786206.367626 trunc from 0x8525be to 0x35fd71 (0x4f284c bytes) 000101[0] 1790786206.411943 punch from 0x0fbb37 to 0x0fc95d (0xe27 bytes) 000104[1] 1790786206.438696 write 0x5e7046 thru 0x5e7422 (0x3dd bytes) 000105[1] 1790786206.473205 trunc from 0x5e7423 to 0x93d17e (0x355d5c bytes) 000106[1] 1790786206.512108 read 0x8ae344 thru 0x8af878 (0x1535 bytes) 000107[0] 1790786206.538556 mapwrite 0x7aed6f thru 0x7b0d68 (0x1ffa bytes) 000108[1] 1790786206.669629 read 0x6aebf7 thru 0x6afa3e (0xe48 bytes) 000109[1] 1790786206.696271 trunc from 0x93d17f to 0x75028e (0x1ecef0 bytes) 000110[0] 1790786206.770159 trunc from 0x75028f to 0x349568 (0x406d26 bytes) 000112[0] 1790786206.814337 trunc from 0x349569 to 0x330168 (0x19400 bytes) 000114[0] 1790786206.831176 mapwrite 0x2d1c8d thru 0x2d3407 (0x177b bytes) 000115[1] 1790786206.898952 trunc from 0x330169 to 0x37f83e (0x4f6d6 bytes) 000116[1] 1790786206.942461 read 0x2ebc74 thru 0x2ec96b (0xcf8 bytes) 000119[0] 1790786207.080311 read 0x000554 thru 0x0008a9 (0x356 bytes) 000122[0] 1790786207.236084 punch from 0x207aea to 0x20800f (0x526 bytes) 000125[1] 1790786207.305901 read 0x0b3834 thru 0x0b56c7 (0x1e94 bytes) 000127[1] 1790786207.434275 mapwrite 0x4da1a0 thru 0x4da738 (0x599 bytes) 000128[1] 1790786207.675953 write 0x754e1e thru 0x756c11 (0x1df4 bytes) 000131[0] 1790786207.727505 read 0x061bb3 thru 0x0620bc (0x50a bytes) 000134[0] 1790786207.790333 mapread 0x502bc9 thru 0x503ea8 (0x12e0 bytes) 000135[0] 1790786207.829095 write 0x038e4e thru 0x03a3fb (0x15ae bytes) 000136[1] 1790786207.869927 trunc from 0x756c12 to 0x55fdbe (0x1f6e53 bytes) 000138[1] 1790786207.997635 punch from 0x160301 to 0x161e15 (0x1b15 bytes) 000139[0] 1790786208.038294 punch from 0x20e78a to 0x210428 (0x1c9f bytes) 000140[0] 1790786208.089021 write 0x880b9c thru 0x880ee6 (0x34b bytes) 000143[1] 1790786208.177203 mapread 0x83e262 thru 0x83eb26 (0x8c5 bytes) 000144[0] 1790786208.265123 trunc from 0x880ee7 to 0x67809a (0x208e4c bytes) 000145[1] 1790786208.346180 punch from 0x20de32 to 0x20f892 (0x1a61 bytes) 000147[1] 1790786208.409347 trunc from 0x67809b to 0x48841c (0x1efc7e bytes) 000150[1] 1790786208.488129 mapwrite 0x24ca12 thru 0x24ca64 (0x53 bytes) 000151[0] 1790786208.578557 write 0x0baca1 thru 0x0bbb71 (0xed1 bytes) 000152[1] 1790786208.610021 trunc from 0x48841d to 0x3a53e6 (0xe3036 bytes) 000153[1] 1790786208.683104 trunc from 0x3a53e7 to 0x48f8e6 (0xea500 bytes) 000155[1] 1790786208.746281 mapwrite 0x1f2d17 thru 0x1f3c38 (0xf22 bytes) 000158[0] 1790786208.813729 trunc from 0x48f8e7 to 0x8e8bdc (0x4592f6 bytes) 000160[0] 1790786208.942119 mapread 0x615b1c thru 0x6171f7 (0x16dc bytes) 000164[1] 1790786209.068432 trunc from 0x8e8bdd to 0x2505af (0x69862d bytes) 000165[0] 1790786209.160584 write 0x37ffcf thru 0x3812b4 (0x12e6 bytes) 000166[0] 1790786209.226444 mapwrite 0x1b1a7d thru 0x1b27c9 (0xd4d bytes) 000167[1] 1790786209.368214 write 0x7a3740 thru 0x7a49e4 (0x12a5 bytes) 000169[1] 1790786209.421733 mapwrite 0x36ba72 thru 0x36c557 (0xae6 bytes) 000170[1] 1790786209.596762 write 0x67bec7 thru 0x67de32 (0x1f6c bytes) 000171[0] 1790786209.648914 trunc from 0x7a49e5 to 0x8371f6 (0x92812 bytes) 000173[1] 1790786209.712065 mapread 0x787590 thru 0x7890c1 (0x1b32 bytes) 000179[1] 1790786209.868029 punch from 0x323420 to 0x323d5a (0x93b bytes) 000180[0] 1790786209.907529 read 0x5c4726 thru 0x5c58b3 (0x118e bytes) 000181[0] 1790786209.914196 mapread 0x5b075b thru 0x5b0c71 (0x517 bytes) 000184[0] 1790786209.941264 trunc from 0x8371f7 to 0x8cbdd5 (0x94bdf bytes) 000186[1] 1790786210.035652 punch from 0x1f107e to 0x1f20c7 (0x104a bytes) 000187[0] 1790786210.167794 mapwrite 0x6f24f7 thru 0x6f2c92 (0x79c bytes) 000189[0] 1790786210.283658 punch from 0x3e2c0b to 0x3e31ff (0x5f5 bytes) 000192[1] 1790786210.354777 mapread 0x0c13bb thru 0x0c27be (0x1404 bytes) 000194[1] 1790786210.398566 trunc from 0x8cbdd6 to 0x34b6fa (0x5806db bytes) 000195[0] 1790786210.464708 trunc from 0x34b6fb to 0x55ab5f (0x20f465 bytes) 000197[0] 1790786210.528166 write 0x7bcc9b thru 0x7bd3d6 (0x73c bytes) 000198[0] 1790786210.548752 write 0x1c026d thru 0x1c092c (0x6c0 bytes) 000199[1] 1790786210.554209 trunc from 0x7bd3d7 to 0x373715 (0x449cc1 bytes) 000200[0] 1790786210.606910 read 0x1ed26c thru 0x1ed308 (0x9d bytes) 000203[1] 1790786210.717534 trunc from 0x373716 to 0x309bb7 (0x69b5e bytes) 000205[0] 1790786210.777284 punch from 0x1cb13d to 0x1ccacd (0x1991 bytes) 000207[1] 1790786210.819805 mapwrite 0x43d401 thru 0x43ec52 (0x1852 bytes) 000208[0] 1790786210.938181 mapwrite 0x18c250 thru 0x18e1cb (0x1f7c bytes) 000209[1] 1790786211.060409 mapwrite 0x440df7 thru 0x4411bf (0x3c9 bytes) 000211[1] 1790786211.181551 read 0x177311 thru 0x177f53 (0xc43 bytes) 000212[1] 1790786211.226321 punch from 0x3a8ad8 to 0x3a9399 (0x8c2 bytes) 000213[1] 1790786211.247001 mapwrite 0x9f1622 thru 0x9f1fdb (0x9ba bytes) 000214[1] 1790786211.353497 read 0x85c758 thru 0x85d6c4 (0xf6d bytes) 000215[0] 1790786211.382244 mapread 0x17c9f6 thru 0x17e82c (0x1e37 bytes) 000216[1] 1790786211.441428 mapread 0x1d0b94 thru 0x1d12fa (0x767 bytes) 000218[0] 1790786211.486382 read 0x351855 thru 0x352520 (0xccc bytes) 000220[0] 1790786211.562074 read 0x1e5c51 thru 0x1e60ee (0x49e bytes) 000222[1] 1790786211.597505 punch from 0x67897e to 0x679c9f (0x1322 bytes) 000224[1] 1790786211.607147 punch from 0x2c2df1 to 0x2c35ec (0x7fc bytes) 000225[0] 1790786211.651321 read 0x0c42e2 thru 0x0c5ca2 (0x19c1 bytes) truncating to largest ever: 0x9fd9c2 000226[1] 1790786211.684466 trunc from 0x9f1fdc to 0x9fd9c1 (0xb9e6 bytes) 000227[1] 1790786211.728548 punch from 0x3c2807 to 0x3c2c54 (0x44e bytes) 000229[0] 1790786211.746132 mapwrite 0x7fa59e thru 0x7fb195 (0xbf8 bytes) 000231[1] 1790786211.950325 mapwrite 0x3d992a thru 0x3d9fda (0x6b1 bytes) 000232[0] 1790786212.069314 mapwrite 0x48bf0e thru 0x48c5d0 (0x6c3 bytes) 000233[0] 1790786212.207111 mapread 0x181728 thru 0x18226f (0xb48 bytes) 000234[0] 1790786212.247596 mapread 0x372d52 thru 0x373b0e (0xdbd bytes) 000235[1] 1790786212.260135 punch from 0x02f782 to 0x030707 (0xf86 bytes) 000237[1] 1790786212.337170 punch from 0x47d3bf to 0x47da04 (0x646 bytes) 000238[1] 1790786212.362694 mapwrite 0x9007ab thru 0x900881 (0xd7 bytes) 000240[0] 1790786212.461857 punch from 0x4ffcc6 to 0x501b38 (0x1e73 bytes) 000241[0] 1790786212.491905 write 0x8c2f42 thru 0x8c4ab6 (0x1b75 bytes) 000242[1] 1790786212.544251 mapwrite 0x5eaff2 thru 0x5ebb98 (0xba7 bytes) 000243[0] 1790786212.645428 mapwrite 0x0d2626 thru 0x0d2db9 (0x794 bytes) 000244[0] 1790786212.741324 punch from 0x8606e0 to 0x86257c (0x1e9d bytes) 000246[0] 1790786212.782088 punch from 0x834f14 to 0x835fa8 (0x1095 bytes) 000247[1] 1790786212.811726 read 0x8ccc6c thru 0x8ce94d (0x1ce2 bytes) 000248[1] 1790786212.896206 read 0x3840bd thru 0x384b77 (0xabb bytes) 000252[1] 1790786212.976147 write 0x80377f thru 0x805281 (0x1b03 bytes) 000253[0] 1790786213.028857 trunc from 0x9fd9c2 to 0x665d2a (0x397c97 bytes) 000254[1] 1790786213.118299 mapwrite 0x523b6e thru 0x523d31 (0x1c4 bytes) 000255[1] 1790786213.246323 write 0x1f018e thru 0x1f1c38 (0x1aab bytes) 000256[0] 1790786213.302226 mapwrite 0x4f8592 thru 0x4fa267 (0x1cd6 bytes) 000260[0] 1790786213.609419 write 0x2eb4bd thru 0x2ed3a8 (0x1eec bytes) 000262[0] 1790786213.688310 read 0x25702d thru 0x2576e2 (0x6b6 bytes) 000264[0] 1790786213.772399 trunc from 0x665d2b to 0x4b899c (0x1ad38e bytes) 000266[1] 1790786213.844611 mapwrite 0x844f14 thru 0x846a5f (0x1b4c bytes) 000268[0] 1790786214.013422 read 0x18c0a5 thru 0x18d950 (0x18ac bytes) 000269[0] 1790786214.036054 write 0x48b519 thru 0x48ce28 (0x1910 bytes) 000270[1] 1790786214.055902 mapread 0x2e048f thru 0x2e19d5 (0x1547 bytes) 000273[1] 1790786214.090603 punch from 0x120617 to 0x121ce7 (0x16d1 bytes) 000274[1] 1790786214.111651 trunc from 0x846a60 to 0x552e67 (0x2f3bf8 bytes) 000275[0] 1790786214.179410 mapread 0x36eab8 thru 0x36ff8a (0x14d3 bytes) 000278[1] 1790786214.265407 read 0x152f17 thru 0x152f4b (0x35 bytes) 000281[0] 1790786214.279886 trunc from 0x552e68 to 0x83f284 (0x2ec41d bytes) 000282[0] 1790786214.364435 punch from 0x1b700b to 0x1b777c (0x772 bytes) 000284[1] 1790786214.369737 write 0x259ffe thru 0x25ac34 (0xc37 bytes) 000287[0] 1790786214.455135 write 0x569040 thru 0x569062 (0x23 bytes) 000289[1] 1790786214.503305 punch from 0x6affa4 to 0x6b104c (0x10a9 bytes) 000290[0] 1790786214.533790 trunc from 0x83f285 to 0x1ec75c (0x652b28 bytes) 000291[1] 1790786214.621336 read 0x1615a9 thru 0x162a6b (0x14c3 bytes) 000295[1] 1790786214.732731 mapread 0x12bc26 thru 0x12bf8a (0x365 bytes) 000297[0] 1790786214.746253 mapwrite 0x699c58 thru 0x69b48d (0x1836 bytes) 000299[1] 1790786214.880128 mapwrite 0x31a723 thru 0x31b428 (0xd06 bytes) 000301[0] 1790786215.117964 mapread 0x545605 thru 0x546a65 (0x1461 bytes) 000304[1] 1790786215.179443 mapread 0x18fe0f thru 0x190c9e (0xe90 bytes) 000305[0] 1790786215.213179 read 0x373be3 thru 0x375170 (0x158e bytes) 000307[1] 1790786215.255880 punch from 0x1e1c97 to 0x1e2cbf (0x1029 bytes) 000310[0] 1790786215.328659 trunc from 0x69b48e to 0x9c214a (0x326cbd bytes) 000311[1] 1790786215.395083 trunc from 0x9c214b to 0x067c3b (0x95a50f bytes) 000312[1] 1790786215.482209 trunc from 0x067c3c to 0x0e0fe5 (0x793aa bytes) 000314[0] 1790786215.549128 write 0x6c4b23 thru 0x6c65ee (0x1acc bytes) 000316[0] 1790786215.618873 punch from 0x18534c to 0x186445 (0x10fa bytes) 000317[1] 1790786215.637012 mapwrite 0x800a60 thru 0x800f36 (0x4d7 bytes) 000321[1] 1790786215.863519 mapread 0x32b0d2 thru 0x32b72f (0x65e bytes) 000322[1] 1790786215.908629 punch from 0x2f9f35 to 0x2fbc0c (0x1cd8 bytes) 000326[0] 1790786215.914746 write 0x939cde thru 0x93a11d (0x440 bytes) 000327[0] 1790786215.954419 punch from 0x3d9f7a to 0x3dad30 (0xdb7 bytes) 000328[0] 1790786215.972080 write 0x248871 thru 0x24a087 (0x1817 bytes) 000331[0] 1790786216.033219 trunc from 0x93a11e to 0x9d62ba (0x9c19d bytes) 000334[1] 1790786216.117083 read 0x1addfb thru 0x1af374 (0x157a bytes) 000335[0] 1790786216.141288 trunc from 0x9d62bb to 0x7379d1 (0x29e8e9 bytes) 000336[1] 1790786216.209177 mapread 0x28a133 thru 0x28b53c (0x140a bytes) 000337[1] 1790786216.290581 mapwrite 0x81c2ab thru 0x81d24d (0xfa3 bytes) 000338[0] 1790786216.404705 mapwrite 0x142ddd thru 0x1447fc (0x1a20 bytes) 000339[0] 1790786216.543808 punch from 0x00121e to 0x002be0 (0x19c3 bytes) 000341[1] 1790786216.576581 punch from 0x1b89ac to 0x1b8aa4 (0xf9 bytes) 000342[1] 1790786216.596292 mapread 0x46648d thru 0x466d5a (0x8ce bytes) 000343[0] 1790786216.602227 punch from 0x09dc07 to 0x09e93e (0xd38 bytes) 000346[1] 1790786216.670744 trunc from 0x81d24e to 0x59a68d (0x282bc0 bytes) 000347[1] 1790786216.711200 trunc from 0x59a68e to 0x6de71c (0x14408f bytes) 000348[0] 1790786216.766534 punch from 0x4fd5a6 to 0x4ff3f8 (0x1e53 bytes) 000349[0] 1790786216.814610 mapwrite 0x27da91 thru 0x27ee9a (0x140a bytes) 000350[1] 1790786217.004543 read 0x66d695 thru 0x66e65a (0xfc6 bytes) 000354[0] 1790786217.121164 read 0x494015 thru 0x495ab9 (0x1aa5 bytes) 000355[0] 1790786217.210887 mapread 0x120b04 thru 0x1223f9 (0x18f6 bytes) 000356[0] 1790786217.338646 mapwrite 0x688511 thru 0x689e6a (0x195a bytes) 000360[1] 1790786217.561441 write 0x920d98 thru 0x921e34 (0x109d bytes) 000361[0] 1790786217.594794 mapwrite 0x5fd5f6 thru 0x5ff525 (0x1f30 bytes) 000362[1] 1790786217.677723 write 0x02d370 thru 0x02eaf3 (0x1784 bytes) 000363[0] 1790786217.715063 read 0x3be609 thru 0x3bed26 (0x71e bytes) 000364[0] 1790786217.763591 punch from 0x0dfb4e to 0x0dfc3e (0xf1 bytes) 000365[0] 1790786217.813222 punch from 0x58c408 to 0x58c923 (0x51c bytes) 000367[0] 1790786217.854411 mapread 0x01433c thru 0x015d0c (0x19d1 bytes) 000368[1] 1790786217.896295 mapwrite 0x1feb04 thru 0x1ffa55 (0xf52 bytes) 000369[1] 1790786218.044416 trunc from 0x921e35 to 0x1f8a1f (0x729415 bytes) 000374[0] 1790786218.168535 write 0x237ac4 thru 0x2395cf (0x1b0c bytes) 000376[1] 1790786218.214371 mapread 0x0d6e90 thru 0x0d773e (0x8af bytes) 000378[0] 1790786218.261256 mapwrite 0x103043 thru 0x103881 (0x83f bytes) 000380[0] 1790786218.420701 read 0x0dd8ba thru 0x0df766 (0x1ead bytes) 000382[0] 1790786218.459040 trunc from 0x2395d0 to 0x062d37 (0x1d6898 bytes) 000383[0] 1790786218.548026 write 0x388860 thru 0x38982e (0xfcf bytes) 000384[0] 1790786218.601260 write 0x2358b4 thru 0x2376ef (0x1e3c bytes) 000385[0] 1790786218.738135 mapread 0x06a6b1 thru 0x06b933 (0x1283 bytes) 000386[1] 1790786218.783730 punch from 0x2faacf to 0x2fc787 (0x1cb9 bytes) 000387[0] 1790786218.858755 write 0x924d9c thru 0x925066 (0x2cb bytes) 000388[1] 1790786218.905997 mapread 0x7e72a2 thru 0x7e7488 (0x1e7 bytes) 000390[0] 1790786219.026951 punch from 0x2db109 to 0x2dc371 (0x1269 bytes) 000392[1] 1790786219.093808 mapread 0x168e72 thru 0x169fa4 (0x1133 bytes) 000394[0] 1790786219.170004 mapread 0x10b224 thru 0x10c176 (0xf53 bytes) 000395[0] 1790786219.231905 mapread 0x57f948 thru 0x581403 (0x1abc bytes) 000396[1] 1790786219.316573 mapread 0x30315a thru 0x30454d (0x13f4 bytes) 000397[1] 1790786219.330238 mapwrite 0x6bc7bc thru 0x6bcec3 (0x708 bytes) 000398[1] 1790786219.457686 mapwrite 0x363247 thru 0x363a18 (0x7d2 bytes) 000401[0] 1790786219.705412 read 0x38607e thru 0x3869e2 (0x965 bytes) 000403[0] 1790786219.765252 punch from 0x47ac75 to 0x47c106 (0x1492 bytes) 000404[0] 1790786219.844958 mapwrite 0x33810d thru 0x338b3b (0xa2f bytes) 000407[0] 1790786220.024836 write 0x1eeacf thru 0x1f03cf (0x1901 bytes) 000408[1] 1790786220.036538 mapread 0x4ac002 thru 0x4adc2d (0x1c2c bytes) 000409[1] 1790786220.076145 read 0x718434 thru 0x71a308 (0x1ed5 bytes) 000413[0] 1790786220.125096 punch from 0x21976f to 0x21b272 (0x1b04 bytes) 000414[0] 1790786220.129996 write 0x0f4efe thru 0x0f5dea (0xeed bytes) 000416[1] 1790786220.173773 trunc from 0x925067 to 0x8ae7c7 (0x7689f bytes) 000421[0] 1790786220.296317 punch from 0x765a6f to 0x7674be (0x1a50 bytes) 000423[0] 1790786220.345726 punch from 0x205c2c to 0x207b40 (0x1f15 bytes) 000424[1] 1790786220.363297 trunc from 0x8ae7c8 to 0x6e244e (0x1cc379 bytes) 000425[0] 1790786220.405055 mapread 0x5ba9d2 thru 0x5bae76 (0x4a5 bytes) 000426[0] 1790786220.455539 punch from 0x5eb11f to 0x5eb9a4 (0x886 bytes) 000428[1] 1790786220.506115 trunc from 0x6e244f to 0x709434 (0x26fe6 bytes) 000429[1] 1790786220.537351 write 0x6868f9 thru 0x6883cd (0x1ad5 bytes) 000430[1] 1790786220.548145 mapread 0x380f20 thru 0x382e55 (0x1f36 bytes) 000431[0] 1790786220.589361 read 0x0f6f0f thru 0x0f83bf (0x14b1 bytes) 000432[1] 1790786220.611838 punch from 0x2d62f7 to 0x2d7a5a (0x1764 bytes) 000433[0] 1790786220.654730 trunc from 0x709435 to 0x81f9bd (0x116589 bytes) 000435[1] 1790786220.702135 punch from 0x2b2c66 to 0x2b4997 (0x1d32 bytes) 000437[0] 1790786220.754390 trunc from 0x81f9be to 0x513c85 (0x30bd38 bytes) 000438[0] 1790786220.787573 mapwrite 0x3e3d53 thru 0x3e4da0 (0x104e bytes) 000440[0] 1790786220.842344 mapwrite 0x4931e0 thru 0x493e8b (0xcac bytes) 000441[0] 1790786220.895537 trunc from 0x513c86 to 0x4c78f2 (0x4c393 bytes) 000442[1] 1790786220.915091 read 0x3cf82a thru 0x3d095b (0x1132 bytes) 000443[1] 1790786220.947117 trunc from 0x4c78f3 to 0x133aba (0x393e38 bytes) 000446[1] 1790786221.020940 read 0x07b148 thru 0x07c1ff (0x10b8 bytes) 000447[0] 1790786221.072157 write 0x42166a thru 0x422bde (0x1575 bytes) 000448[0] 1790786221.100170 mapwrite 0x0c1a9e thru 0x0c2448 (0x9ab bytes) 000449[1] 1790786221.222196 mapread 0x013f2d thru 0x014b9f (0xc73 bytes) 000450[0] 1790786221.281369 punch from 0x0a031c to 0x0a0f1d (0xc02 bytes) 000452[0] 1790786221.334809 mapread 0x191c6a thru 0x193b40 (0x1ed7 bytes) 000453[0] 1790786221.405198 mapread 0x0d23ec thru 0x0d37f5 (0x140a bytes) 000456[0] 1790786221.481108 trunc from 0x422bdf to 0x632cac (0x2100ce bytes) 000457[0] 1790786221.514397 mapwrite 0x325ad8 thru 0x327415 (0x193e bytes) 000458[0] 1790786221.639166 read 0x484475 thru 0x485e63 (0x19ef bytes) 000460[0] 1790786221.711775 punch from 0x5e5490 to 0x5e5abb (0x62c bytes) 000461[0] 1790786221.743718 write 0x3692ce thru 0x369835 (0x568 bytes) 000463[1] 1790786221.795977 mapwrite 0x45ca66 thru 0x45e334 (0x18cf bytes) 000464[1] 1790786221.991796 read 0x3b7bfc thru 0x3b8abb (0xec0 bytes) 000469[0] 1790786222.045318 punch from 0x0ee67b to 0x0f0303 (0x1c89 bytes) 000470[0] 1790786222.071342 mapread 0x38e6ca thru 0x38ee8b (0x7c2 bytes) 000471[1] 1790786222.121247 mapwrite 0x9bd050 thru 0x9be89a (0x184b bytes) 000473[1] 1790786222.352558 mapread 0x540a8e thru 0x54160e (0xb81 bytes) 000474[1] 1790786222.412508 trunc from 0x9be89b to 0x01d71f (0x9a117b bytes) 000475[1] 1790786222.467553 read 0x010564 thru 0x011dd7 (0x1874 bytes) 000476[0] 1790786222.493685 mapwrite 0x97daca thru 0x97f8ad (0x1de4 bytes) 000477[1] 1790786222.648150 read 0x1152c0 thru 0x116c1a (0x195b bytes) 000478[0] 1790786222.700937 mapread 0x31f65a thru 0x320abe (0x1465 bytes) 000483[1] 1790786222.789441 trunc from 0x97f8ae to 0x4c96b4 (0x4b61f9 bytes) 000484[0] 1790786222.856099 mapread 0x45cf77 thru 0x45d756 (0x7e0 bytes) 000485[0] 1790786222.929800 read 0x33e9d8 thru 0x33f950 (0xf79 bytes) 000486[0] 1790786222.954398 write 0x75288f thru 0x7541e6 (0x1958 bytes) 000487[0] 1790786222.976173 mapread 0x2f4af0 thru 0x2f6387 (0x1898 bytes) 000489[0] 1790786223.051529 punch from 0x0648c2 to 0x0650bd (0x7fc bytes) 000490[1] 1790786223.076558 trunc from 0x7541e7 to 0x41e015 (0x3361d1 bytes) 000493[0] 1790786223.168347 mapwrite 0x9f0402 thru 0x9f0db6 (0x9b5 bytes) 000495[0] 1790786223.260677 trunc from 0x9f0db7 to 0x802ad8 (0x1ee2de bytes) 000497[0] 1790786223.325850 mapread 0x426f7a thru 0x426ff3 (0x7a bytes) All operations completed A-OK! PASS 16b (32s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 12:37:08 (1790786228) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (4s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 12:37:13 (1790786233) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.574158 s, 14.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.345428 s, 24.3 MB/s ldlm.namespaces.MGC192.168.203.119@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9eac1008c800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff9eac1223c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff9eac1223c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff9eac1223c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9494 s, 9.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.33383 s, 31.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.5975 s, 40.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.30153 s, 16.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.60372 s, 13.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.55727 s, 29.5 MB/s PASS 16d (47s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 12:38:00 (1790786280) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.304951 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115559 s, 9.1 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 12:38:07 (1790786287) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 12:38:35 (1790786315) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 12:39:03 (1790786343) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 12:39:11 (1790786351) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 12:39:18 (1790786358) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.22696 s, 21.4 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.3363 s, 19.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.53854 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.76508 s, 14.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.38294 s, 11.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.92081 s, 13.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.55491 s, 10.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.24865 s, 21.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56758 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.80996 s, 9.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.20373 s, 8.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.41118 s, 10.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.5234 s, 10.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.98353 s, 8.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11827 s, 8.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11208 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.68245 s, 7.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.72829 s, 7.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.75642 s, 7.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (36s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 12:39:55 (1790786395) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Wed Sep 30 12:40:38 EDT 2026 + fsx 2 done: rc=0 Wed Sep 30 12:40:38 EDT 2026 + fsx 3 done: rc=0 Wed Sep 30 12:40:38 EDT 2026 + fsx 4 done: rc=0 Wed Sep 30 12:40:38 EDT 2026 + fsx 5 done: rc=0 Wed Sep 30 12:40:38 EDT 2026 PASS 16k (49s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 12:40:45 (1790786445) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 12:40:54 (1790786454) mmap test1: basic mmap operation (PASS, 0.13773s) mmap test2: MAP_PRIVATE not write back (PASS, 0.082794s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.4364s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.5923s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.16371s) mmap test6: check mmap write/read content on two nodes (PASS, 0.25725s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.3423s) mmap test11: multi-thread shared mmap (PASS, 1.2828s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1161s) PASS 18 (26s) == sanityn test 19: test concurrent uncached read races ========================================================================= 12:41:20 (1790786480) oleg319-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 12:41:24 (1790786484) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 12:41:31 (1790786491) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 12:41:37 (1790786497) now is 1790786499 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 new atime is 1790786560 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 12:42:45 (1790786565) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 400 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118000 400 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 12:42:51 (1790786571) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 12:42:57 (1790786577) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (6s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 12:43:03 (1790786583) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (3s) == sanityn test 26a: allow mtime to get older ============ 12:43:06 (1790786586) utime: good mknod times 1790786587 <= 1790786588 <= 1790786588 for /mnt/lustre/f26a utime: good mknod times 1790786587 <= 1790786588 <= 1790786588 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 12:43:13 (1790786593) PASS 26b (10s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 12:43:23 (1790786603) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-30 12:43:24.000000000 -0400 Birth: 2026-09-30 12:43:25.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-09-30 12:43:24.000000000 -0400 Birth: 2026-09-30 12:43:25.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 12:43:34 (1790786614) 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 12:43:42 (1790786622) fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 12:43:49 (1790786629) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.119@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (20s) == sanityn test 31r: open-rename(replace) race =========== 12:44:09 (1790786649) -rw-r--r-- 1 root root 0 Sep 30 12:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 12:44:20 (1790786660) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1205: 43153 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 12:44:27 (1790786667) PASS 31t (6s) == sanityn test 32b: lockless i/o ======================== 12:44:34 (1790786674) SKIP: sanityn test_32b max_nolock_bytes is removed >= 2.17.53 SKIP 32b (3s) == sanityn test 32c: contention detection ================ 12:44:37 (1790786677) ldlm.namespaces.filter-lustre-OST0000_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0001_UUID.contended_locks=2 ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.63701 s, 32.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.18717 s, 44.2 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.09742 s, 25.0 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.32917 s, 22.5 MB/s contention_seconds 4, contention_events 20 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.69899 s, 30.9 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.89373 s, 27.7 MB/s ldlm.namespaces.filter-lustre-OST0000_UUID.contention_events=0 ldlm.namespaces.filter-lustre-OST0001_UUID.contention_events=0 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.05248 s, 49.8 MB/s 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.102 s, 47.6 MB/s PASS 32c (27s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 12:45:06 (1790786706) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 12:45:10 (1790786710) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (3s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 12:45:13 (1790786713) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (3s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 12:45:16 (1790786716) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (3s) == sanityn test 34: no lock timeout under IO ============= 12:45:19 (1790786719) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9eac1008c800.ost_server_uuid in IDLE state after 0 sec PASS 34 (132s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 12:47:32 (1790786852) fail_loc=0x80000317 Race attempt 0 Wait for 50454 50494 for 60 sec... - open/close 664 (time 1790786868.67 total 10.02 last 66.28) - open/close 1335 (time 1790786878.67 total 20.02 last 67.08) - open/close 2025 (time 1790786888.67 total 30.02 last 68.99) - open/close 2683 (time 1790786898.68 total 40.03 last 65.76) - open/close 3381 (time 1790786908.69 total 50.04 last 69.75) total: 4000 open/close in 58.74 seconds: 68.09 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.119@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.203.119@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 12:48:44 (1790786924) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg319-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.73564 s, 30.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.8233 s, 28.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.08422 s, 48.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7493632) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.48875 s, 35.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.01739 s, 51.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(4) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.51523 s, 34.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.16712 s, 44.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.15834 s, 45.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.31668 s, 39.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7495680) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.982151 s, 53.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.00225 s, 26.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7444480) after(7495680) PASS 36 (315s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 12:53:59 (1790787239) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 - create 1197 (time 1790787253.32 total 10.00 last 119.67) - create 2404 (time 1790787263.32 total 20.00 last 120.68) - create 3523 (time 1790787273.33 total 30.01 last 111.85) - create 4675 (time 1790787283.33 total 40.01 last 115.16) - create 5840 (time 1790787293.34 total 50.02 last 116.46) - create 7006 (time 1790787303.34 total 60.02 last 116.57) - create 8222 (time 1790787313.34 total 70.02 last 121.56) - create 9440 (time 1790787323.34 total 80.02 last 121.77) total: 10000 create in 85.22 seconds: 117.34 ops/second PASS 37 (96s) == sanityn test 39a: file mtime does not change after rename ========================================================== 12:55:35 (1790787335) repeat after cancel_lru_locks PASS 39a (7s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 12:55:42 (1790787342) repeat after cancel_lru_locks PASS 39b (9s) == sanityn test 39c: check truncate mtime update ================================================================================ 12:55:51 (1790787351) repeat after cancel_lru_locks PASS 39c (10s) == sanityn test 39d: sync write should update mtime ====== 12:56:01 (1790787361) fail_loc=0x411 fail_loc=0 PASS 39d (7s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 12:56:11 (1790787371) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (19s) == sanityn test 40c: pdirops: link and others ======================================================================== 12:56:29 (1790787389) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (16s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 12:56:45 (1790787405) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (19s) == sanityn test 40e: pdirops: rename and others ======================================================================== 12:57:04 (1790787424) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (18s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 12:57:22 (1790787442) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (14s) == sanityn test 41b: pdirops: create vs create ======================================================================== 12:57:36 (1790787456) open: File exists Conflict PASS 41b (12s) == sanityn test 41c: pdirops: create vs link ======================================================================== 12:57:48 (1790787468) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 12:58:02 (1790787482) Conflict PASS 41d (14s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 12:58:17 (1790787497) Conflict PASS 41e (13s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 12:58:30 (1790787510) Conflict PASS 41f (12s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 12:58:42 (1790787522) Conflict PASS 41g (12s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 12:58:54 (1790787534) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2427: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (12s) == sanityn test 41i: reint_open: create vs create ======== 12:59:06 (1790787546) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1065s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 13:16:52 (1790788612) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (12s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 13:17:04 (1790788624) open: File exists Conflict PASS 42b (15s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 13:17:19 (1790788639) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (14s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 13:17:33 (1790788653) Conflict PASS 42d (14s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 13:17:47 (1790788667) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (17s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 13:18:04 (1790788684) Conflict PASS 42f (16s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 13:18:20 (1790788700) Conflict PASS 42g (14s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 13:18:34 (1790788714) Conflict PASS 42h (15s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 13:18:49 (1790788729) PASS 43a (81s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 13:20:10 (1790788810) Conflict PASS 43b (14s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 13:20:24 (1790788824) Conflict PASS 43c (16s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 13:20:40 (1790788840) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 13:20:55 (1790788855) Conflict PASS 43e (13s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 13:21:08 (1790788868) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 13:21:22 (1790788882) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (13s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 13:21:35 (1790788895) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 13:21:50 (1790788910) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (3s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 13:21:54 (1790788914) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (121s) == sanityn test 43k: unlink vs create ==================== 13:23:56 (1790789036) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1121s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 13:42:36 (1790790156) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Wed Sep 30 13:42:46 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:42:51 (1790790171) open: File exists Conflict PASS 44b (12s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:43:03 (1790790183) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (14s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:43:17 (1790790197) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:43:32 (1790790212) Conflict PASS 44e (20s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:43:52 (1790790232) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:44:07 (1790790247) Conflict PASS 44g (13s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:44:20 (1790790260) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:44:34 (1790790274) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:44:38 (1790790278) PASS 45a (141s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:46:59 (1790790419) Conflict PASS 45b (15s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 13:47:14 (1790790434) Conflict PASS 45c (14s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 13:47:28 (1790790448) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 13:47:41 (1790790461) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 13:47:58 (1790790478) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (13s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 13:48:12 (1790790492) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (12s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 13:48:24 (1790790504) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 13:48:38 (1790790518) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (3s) == sanityn test 45j: read vs rename ====================== 13:48:41 (1790790521) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1268s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 14:09:49 (1790791789) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 14:10:04 (1790791804) open: File exists Conflict PASS 46b (14s) == sanityn test 46c: pdirops: link vs link ======================================================================== 14:10:18 (1790791818) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 14:10:31 (1790791831) Conflict PASS 46d (15s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 14:10:45 (1790791845) Conflict PASS 46e (16s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 14:11:01 (1790791861) Conflict PASS 46f (15s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 14:11:16 (1790791876) Conflict PASS 46g (16s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 14:11:32 (1790791892) Conflict PASS 46h (15s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 14:11:47 (1790791907) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 14:11:51 (1790791911) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 14:11:54 (1790791914) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 14:11:58 (1790791918) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (4s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 14:12:02 (1790791922) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (4s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 14:12:06 (1790791926) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (4s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 14:12:10 (1790791930) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 14:12:14 (1790791934) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 14:12:18 (1790791938) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.118308 s, 86.6 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (14s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 14:12:31 (1790791951) 0+1 records in 0+1 records out 158 bytes copied, 0.0308862 s, 5.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3690: kill: (218841) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 14:12:41 (1790791961) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0499045 s, 20.5 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128533 s, 8.2 MB/s 3145728 PASS 51b (24s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 14:13:05 (1790791985) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0282663 s, 36.2 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 14:13:19 (1790791999) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537161 s, 19.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 14:13:27 (1790792007) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2454 0x996 0x240000400 read lease(1) has applied. PASS 51e (11s) == sanityn test 54: rename locking ======================= 14:13:38 (1790792018) llite.lustre-ffff9eac1008c800.xattr_cache=1 llite.lustre-ffff9eac1223c000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505015 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:13:41.000000000 -0400 Modify: 2026-09-30 14:13:41.000000000 -0400 Change: 2026-09-30 14:13:41.000000000 -0400 Birth: 2026-09-30 14:13:41.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505018 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:13:48.000000000 -0400 Modify: 2026-09-30 14:13:49.000000000 -0400 Change: 2026-09-30 14:13:49.000000000 -0400 Birth: 2026-09-30 14:13:48.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505021 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:13:55.000000000 -0400 Modify: 2026-09-30 14:13:55.000000000 -0400 Change: 2026-09-30 14:13:55.000000000 -0400 Birth: 2026-09-30 14:13:55.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:13:54.000000000 -0400 Modify: 2026-09-30 14:13:56.000000000 -0400 Change: 2026-09-30 14:13:56.000000000 -0400 Birth: 2026-09-30 14:13:55.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505024 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:14:01.000000000 -0400 Modify: 2026-09-30 14:14:02.000000000 -0400 Change: 2026-09-30 14:14:02.000000000 -0400 Birth: 2026-09-30 14:14:02.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 16384 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505023 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 14:14:01.000000000 -0400 Modify: 2026-09-30 14:14:02.000000000 -0400 Change: 2026-09-30 14:14:02.000000000 -0400 Birth: 2026-09-30 14:14:02.000000000 -0400 PASS 54 (36s) == sanityn test 55a: rename vs unlink target dir ========= 14:14:14 (1790792054) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (14s) == sanityn test 55b: rename vs unlink source dir ========= 14:14:28 (1790792068) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 14:14:43 (1790792083) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 14:15:04 (1790792104) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 14:15:20 (1790792120) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 14:15:24 (1790792124) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (20s) == sanityn test 55g: rename: race with trylock =========== 14:15:44 (1790792144) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (24s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 14:16:08 (1790792168) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1790792170 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.469531 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (138s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 14:18:27 (1790792307) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1790792314 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.00061 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (139s) == sanityn test 60: Verify data_version behaviour ======== 14:20:46 (1790792446) PASS 60 (10s) == sanityn test 70a: cd directory && rm directory ======== 14:20:56 (1790792456) PASS 70a (9s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 14:21:05 (1790792465) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 14:21:14 (1790792474) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0169145 s, 3.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 14:21:18 (1790792478) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0229516 s, 2.9 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 14:21:23 (1790792483) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 14:21:27 (1790792487) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0235895 s, 2.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 14:21:31 (1790792491) llite.lustre-ffff9eac1008c800.xattr_cache=1 llite.lustre-ffff9eac1223c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 14:21:39 (1790792499) llite.lustre-ffff9eac1008c800.xattr_cache=1 llite.lustre-ffff9eac1223c000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 14:21:46 (1790792506) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0336741 s, 30.4 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0199546 s, 51.3 kB/s 231669: get lock2 231670: get lock1 231669: done 231669: get lock1 231670: done 231670: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 231670: cannot get lock2: Resource deadlock avoided 231670: exit rc=0 231669: done 231669: exit rc=0 PASS 74 (13s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 14:21:59 (1790792519) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129875 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547844 s, 19.1 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4373: 232251 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (15s) == sanityn test 76: Verify MDT open_files listing ======== 14:22:14 (1790792534) opening files: ........ closing files: ........ PASS 76 (76s) == sanityn test 77a: check FIFO NRS policy =============== 14:23:30 (1790792610) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.478176 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934173 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684235 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183025 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149332 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2109 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272822 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274053 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248577 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247246 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244151 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277675 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333046 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297014 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304533 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368498 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341636 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815228 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142558 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145748 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185179 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169572 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00263928 s, 397 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201738 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932923 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777394 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866665 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208825 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035174 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269224 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356527 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285099 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305794 s, 34.3 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 14:23:40 (1790792620) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.442827 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991526 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116797 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807298 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173859 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220671 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261446 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236377 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274466 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274975 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251027 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269946 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307328 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254684 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200495 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260369 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280605 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286348 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365456 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394421 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509587 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795665 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0748071 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412947 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641449 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058549 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693306 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712853 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788949 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735488 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.072828 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814004 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110986 s, 9.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.527954 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055184 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108082 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106307 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167908 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198104 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153626 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219689 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269799 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252744 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2732 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252631 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.303543 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310529 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357199 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374852 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340814 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591116 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502359 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829503 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440137 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608307 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756146 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701626 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987428 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819268 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764409 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882382 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102021 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948004 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650433 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867868 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108031 s, 9.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 14:23:53 (1790792633) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.552416 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08141 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184835 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187749 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15466 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227746 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40146 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311107 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368065 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262915 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304328 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373035 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366763 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410118 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.445291 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374473 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.498769 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149331 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722155 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00928315 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138296 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205557 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179736 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323915 s, 32.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195639 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206227 s, 5.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132436 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22183 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745054 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141599 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225316 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239009 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212062 s, 4.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5768 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896715 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148517 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118707 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118928 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13346 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161579 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223466 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236039 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263834 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266693 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325754 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306099 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359795 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316322 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358637 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317218 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911659 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.052348 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901683 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621442 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0555378 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686358 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676642 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711936 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770075 s, 13.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771789 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078178 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145436 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868507 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126226 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877586 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862581 s, 12.2 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.465792 s, 36.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775049 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107769 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186558 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217569 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262611 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276916 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26434 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222053 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334815 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22811 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247271 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239176 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318376 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30273 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320644 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401842 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719103 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242674 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030913 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00766366 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629838 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177766 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2164 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237893 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953529 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582466 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479339 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708722 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858463 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486804 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229344 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032345 s, 32.4 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 14:24:11 (1790792651) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.313061 s, 53.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043637 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505343 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18871 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183072 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207631 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238969 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253751 s, 4.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215165 s, 4.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302702 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302436 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300264 s, 3.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310139 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268891 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300366 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289617 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322286 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582227 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00255909 s, 410 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00226793 s, 462 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00946409 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022114 s, 474 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784521 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121953 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00816677 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00752018 s, 139 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1876 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127722 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174382 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155754 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659143 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201261 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195539 s, 5.4 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.471904 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611082 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669821 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770251 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147947 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142783 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186541 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141844 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.099051 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169459 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226228 s, 4.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255429 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251364 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33341 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382708 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34589 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354513 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394428 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191213 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271574 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390965 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0394577 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411636 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379862 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0433822 s, 24.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0448472 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045263 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958541 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519186 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861103 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0603066 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628476 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685232 s, 15.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632527 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462968 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101291 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875792 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131815 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184622 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14636 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118198 s, 8.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817078 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110167 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174287 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106547 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211465 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252594 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168759 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1013 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208481 s, 5.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112919 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983715 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147923 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131613 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646507 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683638 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0974385 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0749936 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105775 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813513 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189332 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169764 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985902 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113491 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146735 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125782 s, 8.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=2 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (16s) == sanityn test 77e: check TBF NID nrs policy ============ 14:24:28 (1790792668) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.19@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.359231 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936719 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129865 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226176 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347736 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336203 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136096 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121066 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194349 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463172 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135795 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142266 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124904 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42764 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453182 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136756 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251567 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.489178 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45289 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471315 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121492 s, 86.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22973 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474931 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169906 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010312 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447963 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.501281 s, 2.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.502257 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102257 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.517137 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014372 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181375 s, 57.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.574675 s, 1.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.606515 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083977 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0647153 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285891 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200275 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18475 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161231 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175307 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264018 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256378 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237876 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255972 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280149 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26308 s, 4.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40016 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376936 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401176 s, 2.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580127 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061585 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153962 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635589 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128647 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145679 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069592 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646879 s, 16.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786443 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124962 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14959 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0878166 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122632 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886423 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939267 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104975 s, 10.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.654801 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903967 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132377 s, 7.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160564 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184369 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309227 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29052 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.442265 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.468676 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387268 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.460791 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.505446 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459214 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47282 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237673 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202175 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.560668 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05676 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465451 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00592387 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.541409 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00245444 s, 427 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699969 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00268213 s, 391 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215942 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00263271 s, 398 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00244008 s, 430 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100904 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979784 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083905 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09766 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408998 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209183 s, 50.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65267 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864226 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730411 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988537 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154733 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140404 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132084 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239226 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206657 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227761 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230369 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228574 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298645 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272896 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283126 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368279 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306405 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393567 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046472 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0361706 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736948 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0572542 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212936 s, 49.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103472 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676533 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247134 s, 42.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116718 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742972 s, 14.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103596 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719913 s, 14.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834129 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086892 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764689 s, 13.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (28s) == sanityn test 77f: check TBF JobID nrs policy ========== 14:24:56 (1790792696) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.612503 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084048 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100507 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208425 s, 5.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13248 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192781 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131308 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310451 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240649 s, 4.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322241 s, 3.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306642 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281566 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377908 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364223 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344813 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428084 s, 2.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431815 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12752 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126236 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134863 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773615 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.081386 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902882 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152522 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160936 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110165 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109153 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108264 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114862 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115843 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11691 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123603 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131541 s, 8.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.423012 s, 39.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695778 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571205 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913458 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073512 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873588 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09905 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873039 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826712 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0896154 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112671 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112129 s, 9.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236689 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253847 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243259 s, 4.3 MB/srunning as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265261 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.312024 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191008 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521829 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194033 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553189 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988431 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328697 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327856 s, 32.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344981 s, 30.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00975592 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367111 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0038694 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00720928 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158864 s, 66.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00438125 s, 239 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582049 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147118 s, 71.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.403291 s, 41.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931851 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174927 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121541 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582078 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135319 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2815 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221452 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222585 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322493 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294583 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.414562 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434513 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424452 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262088 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391484 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250429 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827347 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718053 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307372 s, 34.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328925 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408712 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774846 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127037 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546314 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936799 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594974 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10711 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597316 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141266 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.068621 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742662 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0778195 s, 13.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.667757 s, 25.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925223 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115476 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166883 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17603 s, 6.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215937 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248893 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187238 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27697 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208554 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218265 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367195 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336656 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345676 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362292 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0795036 s, 13.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342961 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106676 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614705 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042656 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478274 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289552 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0392487 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0455252 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062604 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941623 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721923 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814741 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985762 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236154 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104117 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12205 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126635 s, 8.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 14:25:22 (1790792722) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (11s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 14:25:34 (1790792734) oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 14:25:47 (1790792747) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (24s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 14:26:11 (1790792771) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.856117 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635982 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888094 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871561 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832961 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169443 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112287 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145004 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217228 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212081 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19757 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234046 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345756 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32462 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342218 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427655 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444029 s, 2.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189278 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215627 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243191 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181653 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185394 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195194 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196569 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197605 s, 5.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205464 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209076 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206852 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211708 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215107 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218805 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222449 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324047 s, 3.2 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.813 s, 9.7 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.622 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (72s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 14:27:23 (1790792843) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29287 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151042 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101898 s, 10.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969985 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149595 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223648 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171373 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159112 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336212 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265903 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27512 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353189 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233673 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255279 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380042 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380354 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282466 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758063 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727016 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776195 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891578 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855297 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951379 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105278 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106946 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897335 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112171 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115349 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102114 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11988 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103944 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977387 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132151 s, 7.9 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0001 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9707 s, 5.3 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29767 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806737 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0452231 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112891 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121743 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199071 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187789 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202756 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201787 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205993 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215573 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207697 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254326 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241007 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167865 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239381 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872999 s, 12.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630209 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670626 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563105 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614134 s, 17.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757761 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775863 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0671318 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864504 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825517 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666809 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808111 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821789 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790161 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08711 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922872 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12272 s, 8.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9759 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9759 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (153s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 14:29:57 (1790792997) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31052 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939197 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161815 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140738 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229851 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171612 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182174 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469235 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279036 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195632 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337358 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100202 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232705 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199659 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300871 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302985 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325803 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105161 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104621 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932195 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101343 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117026 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872071 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112462 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106614 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131173 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149872 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116085 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122285 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14112 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112308 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132482 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143864 s, 7.3 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9652 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9668 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30284 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126268 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132769 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127079 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138557 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158039 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182217 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146665 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165727 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119303 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158846 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233132 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187807 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309416 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270713 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249179 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297881 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761541 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797087 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570662 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850079 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824827 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549721 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586598 s, 17.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836094 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760814 s, 13.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697172 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998864 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548381 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712694 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879631 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826005 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125371 s, 8.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9648 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9281 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (155s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 14:32:31 (1790793151) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.568322 s, 29.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113026 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106412 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115047 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208933 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153776 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173584 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185906 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238204 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179295 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237638 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104703 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169331 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224917 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236117 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24202 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347957 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257923 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249074 s, 42.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197856 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551762 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360589 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0378567 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0622194 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393967 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104486 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551931 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584049 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683873 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668182 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732566 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033174 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119871 s, 87.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.32992 s, 12.6 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.91698 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.19@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.19@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.733476 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289867 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0719475 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0740135 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947219 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559042 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656264 s, 16.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1342 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122045 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815798 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745509 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143418 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226016 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185357 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274093 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265756 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200969 s, 5.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137795 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132271 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140864 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13335 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173856 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137882 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139379 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167374 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135874 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141005 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140069 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184725 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148323 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195941 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166265 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165717 s, 6.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.78858 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0039 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.19@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.695789 s, 24.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611992 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0641596 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747339 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15868 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198443 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2641 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250004 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244647 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306652 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293094 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330531 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345277 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337165 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32813 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333857 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347981 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045544 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0497984 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483293 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100333 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0855903 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596076 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581805 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061797 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599855 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0630968 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733819 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0816183 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861576 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922287 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961456 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101484 s, 10.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.3828 s, 10.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.26903 s, 19.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.673491 s, 24.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850322 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813541 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814033 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566828 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112457 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789593 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18533 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196225 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21818 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252935 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269979 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22795 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227253 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224576 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243428 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27482 s, 3.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858869 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715096 s, 14.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493993 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0580569 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633545 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646255 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727926 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753797 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947122 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785445 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13693 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109795 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954236 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117306 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153661 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117811 s, 8.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7329 s, 9.8 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88316 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.43943 s, 12.4 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94151 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.07075 s, 15.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773121 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919032 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1348 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954262 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184487 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203574 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2209 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235109 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25505 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24624 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296881 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213922 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233448 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223629 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277112 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331992 s, 3.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177353 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551647 s, 19.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678468 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276228 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569181 s, 18.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852601 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632638 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419959 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818472 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953622 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621433 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0494548 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0633665 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700209 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177871 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196373 s, 53.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7681 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6353 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.550478 s, 30.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107601 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919487 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664265 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821616 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160182 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243817 s, 4.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265842 s, 3.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270472 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293358 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279646 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28923 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216111 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317765 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293922 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338172 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33014 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542076 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387165 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385223 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322589 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597388 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0590855 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505416 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638812 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576052 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0843936 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349404 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722493 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721728 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071439 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056733 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634077 s, 16.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.22334 s, 12.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95458 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.469201 s, 35.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113432 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101789 s, 10.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114216 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187359 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263344 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264903 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349655 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197696 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211855 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243489 s, 4.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.360562 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200021 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368794 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275967 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325839 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305977 s, 3.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550024 s, 19.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582963 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601283 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189225 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892115 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862409 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183688 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00785792 s, 133 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152299 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115948 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996206 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102699 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152499 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316315 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421599 s, 24.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541168 s, 19.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3782 s, 9.2 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9054 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.0363 s, 9.5 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99228 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (438s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 14:39:49 (1790793589) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.813472 s, 20.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109969 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126114 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12477 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150418 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274318 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215571 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21357 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292172 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225913 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418737 s, 2.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37704 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.427141 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344973 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.440028 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.417916 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418389 s, 2.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353518 s, 297 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338515 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246392 s, 426 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00238256 s, 440 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00250764 s, 418 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139699 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186574 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137871 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119839 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0907235 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779513 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435284 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0628472 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0264216 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598457 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188784 s, 55.5 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0648 s, 10.4 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0248 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (54s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 14:40:43 (1790793643) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.19@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.19@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.733005 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931304 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103319 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170565 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181722 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267379 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163745 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188536 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276898 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.320701 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317807 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233102 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222319 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322895 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248072 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345639 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.370857 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134502 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118794 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137719 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956826 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158439 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149338 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149077 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250794 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245895 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256562 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176142 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123747 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115201 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188191 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278117 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206724 s, 5.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=4 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.75659 s, 12.0 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1877 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (56s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 14:41:39 (1790793699) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.19@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.756078 s, 22.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037242 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0483141 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900383 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798499 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123067 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128761 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153655 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147157 s, 7.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176409 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172221 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182467 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186884 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174236 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195121 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187463 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190494 s, 5.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff9eac1008c800.lock_unused_count=4 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0691807 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669808 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0492393 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0502365 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685746 s, 15.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0370482 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781267 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240095 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678886 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583688 s, 18.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984366 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820369 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00500435 s, 210 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0270887 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00668637 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287964 s, 36.4 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.12011 s, 11.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.47382 s, 19.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (48s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 14:42:27 (1790793747) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.754108 s, 22.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662891 s, 15.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123474 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232238 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184196 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14023 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177297 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316742 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258021 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31531 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277104 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237994 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242077 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33497 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403097 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355663 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0792013 s, 13.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11984 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152618 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133047 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961803 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141604 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12766 s, 8.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956073 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159656 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146578 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145068 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154153 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175588 s, 6.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20212 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227965 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188815 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214572 s, 4.9 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3165 s, 9.3 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95123 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.06325 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93073 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (95s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 14:44:02 (1790793842) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0644 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545131 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827039 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774964 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144423 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772833 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144077 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254967 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179759 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165205 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249831 s, 4.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245579 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206374 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202103 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316575 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240687 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289372 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245389 s, 42.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100446 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216858 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352885 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443914 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775986 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11547 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124042 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125755 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550885 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644962 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00383873 s, 273 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00938009 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00700634 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101789 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800859 s, 131 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.885 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0003 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (76s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 14:45:18 (1790793918) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.770868 s, 21.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640721 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0586657 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981604 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188071 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264869 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317655 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167182 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.344941 s, 3.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309659 s, 3.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165961 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130804 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.424252 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.375145 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319689 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac1008c800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214036 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.410177 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339372 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462181 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0447098 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516128 s, 20.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693499 s, 15.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063344 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114472 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0611717 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737369 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825744 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642311 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195645 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929174 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119067 s, 8.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117009 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119171 s, 8.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.24936 s, 12.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99263 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg319-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.710892 s, 23.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850468 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648059 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625666 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186029 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176917 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221388 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197144 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162189 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220109 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22575 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146333 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246811 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207904 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165199 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237762 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253144 s, 4.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506028 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521306 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0768322 s, 13.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591995 s, 17.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651511 s, 16.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525178 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0663672 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638707 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735812 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645715 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722537 s, 14.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897355 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714937 s, 14.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094543 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751036 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821718 s, 12.8 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.7211 s, 9.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.92368 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.4994 s, 7.8 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93749 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (148s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 14:47:47 (1790794067) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg319-client.virtnet /mnt/lustre (opts:) Stopping client oleg319-client.virtnet /mnt/lustre opts: Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg319-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg319-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg319-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.19,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg319-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.19,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg319-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.19,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/71150/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.203.19,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg319-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: kmemleak disabled oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre Starting client oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre Started clients oleg319-client.virtnet: 192.168.203.119@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Starting client oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 Started clients oleg319-client.virtnet: 192.168.203.119@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9eac1009e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9eac18104800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac1009e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac18104800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.538942 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117772 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438732 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654202 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151995 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282902 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237304 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27082 s, 3.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199296 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20042 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381791 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362885 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422429 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390237 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421633 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.471428 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.46582 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9eac18104800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128666 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200348 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15757 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144035 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168507 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151549 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147487 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160001 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160749 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177161 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185223 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187978 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210414 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181481 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217062 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208925 s, 5.0 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.67785 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.29209 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03917 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143703 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118134 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141149 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591601 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199946 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265285 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992078 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105266 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179813 s, 5.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac18104800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48498 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206453 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.521589 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.446902 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328709 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281352 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.581093 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156518 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130835 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138497 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140918 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139996 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148896 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255423 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171266 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171259 s, 6.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167701 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266751 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17222 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202776 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180384 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284321 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299828 s, 3.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=4 removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.439 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.89746 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (207s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 14:51:13 (1790794273) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (18s) == sanityn test 77kj: Verify nodemap support for NRS TBF rule ========================================================== 14:51:31 (1790794291) On MGS 192.168.203.119, active = nodemap.active=1 On MGS 192.168.203.119, TBF = ost.OSS.ost_io.nrs_policies=tbf nodemap ost.OSS.ost_io.nrs_tbf_rule=start ext_nm_rw nodemap={TBF} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.491968 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753429 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770892 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890354 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185671 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214403 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21111 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152691 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161769 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.252479 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209022 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248557 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305198 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261548 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222925 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249098 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23064 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805941 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484951 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050763 s, 20.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299417 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258907 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252189 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308051 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00680621 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0436318 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469123 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673773 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569307 s, 18.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110265 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589158 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474437 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765439 s, 13.7 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.95629 s, 13.2 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.09354 s, 20.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete On MGS 192.168.203.119, TBF.id = On MGS 192.168.203.119, active = nodemap.active=0 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.677401 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879083 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169016 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0963316 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585177 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862511 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1574 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122549 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10705 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121951 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147998 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306078 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322635 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326593 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9eac18104800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300008 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.346629 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357179 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256998 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661247 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0726317 s, 14.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442307 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163897 s, 64.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544695 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295816 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102735 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00715964 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00786256 s, 133 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472185 s, 22.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063529 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108722 s, 96.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214203 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518221 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536537 s, 19.5 MB/s removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg319-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 1 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.16437 s, 12.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.90102 s, 36.1 MB/s Read runtime is 4 s, speed is 25.000000 IOPS read rate (25.000000) should exceed limit (20 * 1) cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_nm_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf nodemap+opcode oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_w nodemap={TBF}&opcode={ost_write} rate=20': Invalid argument oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10: Invalid argument oleg319-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_r nodemap={TBF}&opcode={ost_read} rate=10': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 On MGS 192.168.203.119, active = nodemap.active=1 On MGS 192.168.203.119, TBF = ost.OSS.ost_io.nrs_tbf_rule=start ext_w nodemap={TBF}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nodemap={TBF}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.660464 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694136 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861142 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105064 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18752 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123704 s, 8.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235084 s, 4.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145188 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135116 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217119 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251547 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183911 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165046 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12385 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248186 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315075 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124219 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff9eac18104800.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133784 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157199 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145219 s, 7.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15533 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133468 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133693 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161856 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150855 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139186 s, 7.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150181 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144284 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226596 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162434 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169102 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170403 s, 6.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170602 s, 6.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=1 removed '/mnt/lustre/d77kj.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77kj.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77kj.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.38294 s, 12.5 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.87695 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kj.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo On MGS 192.168.203.119, TBF.id = On MGS 192.168.203.119, active = nodemap.active=0 PASS 77kj (176s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 14:54:27 (1790794467) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (11s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 14:54:38 (1790794478) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg319-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.50838 s, 299 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg319-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0964 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (58s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 14:55:36 (1790794536) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.497646 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050226 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943422 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083661 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145227 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509868 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123172 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150127 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110411 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166977 s, 6.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff9eac18104800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295764 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196394 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304986 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257485 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330494 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379092 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493025 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff9eac18104800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815799 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00579949 s, 181 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722674 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00800389 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00669337 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806847 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00855433 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150692 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934591 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154173 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149094 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725671 s, 14.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153177 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173375 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154126 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581619 s, 18.0 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.63997 s, 10.9 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.25159 s, 20.0 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.420331 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417812 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054159 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605928 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136421 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146739 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189843 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198015 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2155 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234488 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210901 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22904 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227036 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219149 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234048 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237654 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296798 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306913 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853818 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409971 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897562 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675446 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939936 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674388 s, 15.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813029 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827225 s, 12.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108363 s, 9.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857869 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901183 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858773 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979173 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947639 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132427 s, 7.9 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg319-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg319-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' not_under_control: 0 cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.74145 s, 13.5 MB/s Write runtime is 8 s, speed is 12.500000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.23622 s, 20.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (91s) == sanityn test 77o: Changing rank should not panic ====== 14:57:07 (1790794627) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (12s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 14:57:19 (1790794639) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (105s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 14:59:05 (1790794745) mds.MDS.mdt.nrs_policies=tbf oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg319-client: oleg319-server: ssh exited with exit code 36 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg319-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg319-client: oleg319-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (32s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 14:59:36 (1790794776) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 273636 kill md_thread xargs PASS 77r (60s) == sanityn test 77s: Check TBF LRU shrinker ============== 15:00:37 (1790794837) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 9 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 9 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (69s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 15:01:46 (1790794906) oleg319-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg319-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg319-client: oleg319-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (10s) == sanityn test 79: xattr: intent error ================== 15:01:56 (1790794916) File: /mnt/lustre/d79.sanityn Size: 16384 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935687 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 15:01:59.000000000 -0400 Modify: 2026-09-30 15:01:59.000000000 -0400 Change: 2026-09-30 15:01:59.000000000 -0400 Birth: 2026-09-30 15:01:59.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 15:02:16 (1790794936) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 15:02:20 (1790794940) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (3s) == sanityn test 81a: rename and stat under striped directory ========================================================== 15:02:23 (1790794943) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 15:02:27 (1790794947) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 15:02:31 (1790794951) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (3s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 15:02:34 (1790794954) - open/close 620 (time 1790794970.57 total 10.01 last 61.92) - open/close 1295 (time 1790794980.58 total 20.02 last 67.45) - open/close 1960 (time 1790794990.58 total 30.02 last 66.49) total: 2500 open/close in 38.96 seconds: 64.16 ops/second snapshot_time 1790795106.435873688 secs.nsecs start_time 1790794958.663106927 secs.nsecs elapsed_time 147.772766761 secs.nsecs open 2505 samples [usecs] 125 24711 4167557 13960893045 close 2505 samples [usecs] 134 30314 999311 1943188547 mknod 2501 samples [usecs] 503 24608 3614527 11108108623 unlink 2500 samples [usecs] 742 122145 6835426 90131554608 mkdir 2 samples [usecs] 1102 6642 7744 45330568 rmdir 2 samples [usecs] 7146 9827 16973 147635245 rename 2500 samples [usecs] 501 79367 21125601 380766967855 getattr 3763 samples [usecs] 10 16735 371157 1325009635 parallel_rename_file 2500 samples [usecs] 501 79367 21125601 380766967855 crossdir_rename 2500 samples [usecs] 501 79367 21125601 380766967855 readpage 25 samples [usecs] 5598 68185 650460 22046233728 PASS 81d (157s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 15:05:11 (1790795111) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7623 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7623 PASS 82 (7s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 15:05:19 (1790795119) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 15:05:23 (1790795123) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 15:05:38 (1790795138) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1790795140 Finishing test test1 at 1790795145 PASS 85 (13s) == sanityn test 90: open/create and unlink striped directory ========================================================== 15:05:50 (1790795150) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (3s) == sanityn test 91: chmod and unlink striped directory === 15:05:54 (1790795154) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 15:05:57 (1790795157) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 15:06:01 (1790795161) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5846 0x16d6 0x240000400 1 5741 0x166d 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5742 0x166e 0x280000400 0 5847 0x16d7 0x240000400 rc1=0 and rc2=0 PASS 93 (17s) == sanityn test 94: signal vs CP callback race =========== 15:06:18 (1790795178) osc.lustre-OST0000-osc-ffff9eac1009e800.idle_timeout=0 osc.lustre-OST0000-osc-ffff9eac18104800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9eac1009e800.idle_timeout=0 osc.lustre-OST0001-osc-ffff9eac18104800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.207137 s, 39.5 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.02868 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6389: 290845 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.148734 s, 27.5 kB/s osc.lustre-OST0000-osc-ffff9eac1009e800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9eac18104800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac1009e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9eac18104800.idle_timeout=debug PASS 94 (15s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 15:06:34 (1790795194) llite.lustre-ffff9eac1009e800.fast_read=0 llite.lustre-ffff9eac18104800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0453297 s, 46.3 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff9eac1009e800.fast_read=1 llite.lustre-ffff9eac18104800.fast_read=1 PASS 95a (21s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 15:06:54 (1790795214) llite.lustre-ffff9eac1009e800.fast_read=0 llite.lustre-ffff9eac18104800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.023454 s, 524 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff9eac1009e800.fast_read=1 llite.lustre-ffff9eac18104800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 15:07:07 (1790795227) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 15:07:10 (1790795230) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00601543 s, 681 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 15:07:17 (1790795237) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162459 s, 12.9 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (7s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 15:07:24 (1790795244) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257872 s, 8.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 15:07:32 (1790795252) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.0113195 s, 1.1 MB/s PASS 100e (8s) == sanityn test 101a: Discard DoM data on unlink ========= 15:07:40 (1790795260) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0130137 s, 315 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 15:07:48 (1790795268) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0155584 s, 263 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 15:07:55 (1790795275) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00919716 s, 445 kB/s PASS 101c (9s) == sanityn test 102: Test open by handle of unlinked file ========================================================== 15:08:04 (1790795284) file: /mnt/lustre/f102.sanityn fh_bytes: 32 fh_type: 151 fh_data: d1 0b 00 00 02 00 00 00 68 0f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 file's parent FID is [0:0x0:0x0] file FID is [0x200000bd1:0xf68:0x0] access via mount point '/mnt/lustre2' - OK access after unlink - OK check_fhandle_syscalls test Passed! PASS 102 (7s) == sanityn test 103: Test size correctness with lockahead ========================================================== 15:08:11 (1790795291) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1790795295 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1790795298 Finishing test test23 at 1790795299 PASS 103 (15s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 15:08:26 (1790795306) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (4s) == sanityn test 105: Glimpse and lock cancel race ======== 15:08:30 (1790795310) fail_loc=0x416 PASS 105 (30s) == sanityn test 106a: Verify the btime via statx() ======= 15:09:00 (1790795340) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (5s) == sanityn test 106b: Glimpse RPCs test for statx ======== 15:09:05 (1790795345) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0279209 s, 36.7 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 15:09:07.000000000 -0400 Modify: 2026-09-30 15:09:07.000000000 -0400 Change: 2026-09-30 15:09:07.000000000 -0400 Birth: 2026-09-30 15:09:08.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938223 -rw-r--r-- 1 0 0 1790795348 1790795347 1790795347 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938223 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 15:09:07.000000000 -0400 Modify: 2026-09-30 15:09:07.000000000 -0400 Change: 2026-09-30 15:09:07.000000000 -0400 Birth: 2026-09-30 15:09:08.000000000 -0400 1790795347 1024 11 PASS 106b (10s) == sanityn test 106c: Verify statx attributes mask ======= 15:09:15 (1790795355) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 15:09:23 (1790795363) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.186334 s, 56.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5749 0x1675 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 First grouplock blocks second one PASS 107a (11s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 15:09:34 (1790795374) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.335543 s, 31.3 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5856 0x16e0 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7623 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (16s) == sanityn test 108a: lseek: parallel updates ============ 15:09:50 (1790795390) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00752669 s, 1.3 MB/s llite.lustre-ffff9eac1009e800.hybrid_io=0 llite.lustre-ffff9eac18104800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 5.02307 s, 1.7 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 15:10:03 (1790795403) 192.168.203.119@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre (opts:) 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 192.168.203.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 192.168.203.119@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg319-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg319-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: kmemleak disabled oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre2 PASS 109 (47s) == sanityn test 110: do not grant another lock on resend ========================================================== 15:10:50 (1790795450) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 15:10:52.000000000 -0400 Modify: 2026-09-30 15:10:52.000000000 -0400 Change: 2026-09-30 15:10:52.000000000 -0400 Birth: 2026-09-30 15:10:53.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg319-client.virtnet oleg319-client.virtnet PASS 110 (171s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 15:13:42 (1790795622) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 15:13:45 (1790795625) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (3s) == sanityn test 113: check servers of specified fs ======= 15:13:48 (1790795628) lustre-OST0000-osc-ffff9eac18287800 active. lustre-OST0001-osc-ffff9eac18287800 active. lustre-OST0000-osc-ffff9eac18287800 active. lustre-OST0001-osc-ffff9eac18287800 active. MGC192.168.203.119@tcp active. PASS 113 (7s) == sanityn test 114: implicit default LMV inherit ======== 15:13:56 (1790795636) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 15:13:59 (1790795639) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 15:14:03 (1790795643) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (3s) == sanityn test 121: trunc append race =================== 15:14:06 (1790795646) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0112763 s, 1.8 MB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.115705 s, 177 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: directory size is consistent across mounts ========================================================== 15:14:15 (1790795655) PASS 122 (9s) == sanityn test 200: service remains healthy while able to process request ========================================================== 15:14:24 (1790795664) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1779 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1492 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2923 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1824 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2368 s, 0.2 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2248 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2265 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1771 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2152 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2249 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1768 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3052 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3192 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2008 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3053 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.357 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4994 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3485 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg319-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff9eac14013000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9eac14013000.ost_server_uuid in IDLE state after 0 sec PASS 200 (81s) cleanup: ====================================================== == sanityn test complete, duration 10059 sec ============= 15:15:47 (1790795747) === sanityn: start cleanup 15:15:49 (1790795749) === Stopping clients: oleg319-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg319-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 15:20:59 (1790796059) ===