-----============= acceptance-small: sanityn ============----- Fri Jul 19 05:37:44 EDT 2024 excepting tests: 27 28 102 skipping tests SLOW=no: 33a Starting client oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Started clients oleg328-client.virtnet: 192.168.203.128@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) oleg328-client.virtnet: executing check_config_client /mnt/lustre oleg328-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg328-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5884000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012a5e9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5884000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5e9800.idle_timeout=debug disable quota as required oleg328-server: oleg328-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115045 s, 91.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6859] == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 05:37:53 (1721381873) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 05:37:55 (1721381875) -rw-r--r-- 1 root root 0 Jul 19 05:37 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (1s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 05:37:56 (1721381876) -rw-r--r-- 1 root root 0 Jul 19 05:37 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 05:37:58 (1721381878) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 05:37:59 (1721381879) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 05:38:01 (1721381881) total 1 -rwxrwxrwx 1 root root 0 Jul 19 05:37 f2a -rwxrwxrwx 1 root root 0 Jul 19 05:37 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 19 05:37 f2a -rwxrwxrwx 1 root root 0 Jul 19 05:37 f2b running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00296097 s, 173 kB/s PASS 2e (1s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 05:38:02 (1721381882) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 05:38:03 (1721381883) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0829782 s, 25.3 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (1s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 05:38:04 (1721381884) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 05:38:06 (1721381886) PASS 4 (2s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 05:38:08 (1721381888) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 05:38:10 (1721381890) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (1s) == sanityn test 7: remove of open directory on other node ======================================================================= 05:38:11 (1721381891) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 05:38:13 (1721381893) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (1s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 05:38:15 (1721381895) PASS 9a (1s) == sanityn test 9b: append to striped sparse file ======== 05:38:16 (1721381896) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124502 s, 84.2 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0136456 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 05:38:18 (1721381898) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00251616 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00175897 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00200325 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00324552 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00129496 s, 0.8 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00274279 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00185938 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00146789 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00158278 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00139539 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00177648 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00175972 s, 0.6 kB/s PASS 10a (1s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 05:38:19 (1721381899) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00325456 s, 944 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00657551 s, 623 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000230895 s, 13.3 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 05:38:21 (1721381901) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 273: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 05:38:23 (1721381903) warning: '-runtime' deprecated, use '-t runtime' instead - link 1915 (time 1721381913.65 total 10.00 last 191.42) - link 3893 (time 1721381923.65 total 20.01 last 197.78) - link 5925 (time 1721381933.65 total 30.01 last 203.19) - link 7837 (time 1721381943.65 total 40.01 last 191.16) - link 10000 (time 1721381952.62 total 48.97 last 241.25) - link 12151 (time 1721381962.62 total 58.97 last 215.07) - link 14195 (time 1721381972.62 total 68.97 last 204.40) - link 16101 (time 1721381982.62 total 78.98 last 190.58) - link 18178 (time 1721381992.63 total 88.98 last 207.61) - link 20000 (time 1721382001.98 total 98.34 last 194.73) total: 20316 link in 100.00 seconds: 203.16 ops/second using seed 882742171 running for 100 seconds - stat 10000 (time 1721381907 ; total 3 ; last 3) - stat 20000 (time 1721381911 ; total 7 ; last 4) - stat 30000 (time 1721381914 ; total 10 ; last 3) - stat 40000 (time 1721381917 ; total 13 ; last 3) - stat 50000 (time 1721381921 ; total 17 ; last 4) - stat 60000 (time 1721381924 ; total 20 ; last 3) - stat 70000 (time 1721381927 ; total 23 ; last 3) - stat 80000 (time 1721381931 ; total 27 ; last 4) - stat 90000 (time 1721381934 ; total 30 ; last 3) - stat 100000 (time 1721381937 ; total 33 ; last 3) - stat 110000 (time 1721381941 ; total 37 ; last 4) - stat 120000 (time 1721381944 ; total 40 ; last 3) - stat 130000 (time 1721381947 ; total 43 ; last 3) - stat 140000 (time 1721381951 ; total 47 ; last 4) - stat 150000 (time 1721381954 ; total 50 ; last 3) - stat 160000 (time 1721381961 ; total 57 ; last 7) - stat 170000 (time 1721381969 ; total 65 ; last 8) - stat 180000 (time 1721381977 ; total 73 ; last 8) - stat 190000 (time 1721381985 ; total 81 ; last 8) - stat 200000 (time 1721381993 ; total 89 ; last 8) - stat 210000 (time 1721382001 ; total 97 ; last 8) total: 213180 stats in 100 seconds: 2131.800049 stats/second - unlinked 0 (time 1721381954 ; total 0 ; last 0) - unlinked 10000 (time 1721382016 ; total 62 ; last 62) - unlinked 20000 (time 1721382043 ; total 89 ; last 27) unlink(/mnt/lustre2/lockdir/lockfile20316) error: No such file or directory total: 20315 unlinks in 90 seconds: 225.722229 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (20454) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (20457) - No such process PASS 12 (142s) == sanityn test 13: test directory page revocation ======= 05:40:45 (1721382045) f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 05:40:47 (1721382047) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 308: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4253: 21833 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (1s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 05:40:48 (1721382048) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4253: 22460 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (1s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 05:40:50 (1721382050) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 323: 23086 Terminated $DIR1/$tdir/sleep 60 PASS 14b (1s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 05:40:51 (1721382051) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: 23715 Terminated $DIR1/$tdir/sleep 60 PASS 14c (1s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 05:40:52 (1721382052) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 346: 24343 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 05:40:54 (1721382054) PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: STRIPECOUNT=2 ORIGFREE=7517184 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7517184kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed sleep 5 for ZFS zfs Waiting for MDT destroys to complete SKIP 15 (10s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 05:41:04 (1721382064) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0841778 s, 24.9 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0583245 s, 36.0 MB/s Chance of close/open is 1 in 50 Seed set to 2065 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003[0] 1721382065.704078 write 0x1c6985 thru 0x1d5ab9 (0xf135 bytes) 000005[1] 1721382065.714398 mapread 0x0b579e thru 0x0c1d06 (0xc569 bytes) 000010[0] 1721382065.761663 mapread 0x16191c thru 0x16bd82 (0xa467 bytes) 000011[0] 1721382065.772156 mapwrite 0x32e2cc thru 0x33969a (0xb3cf bytes) 000012[0] 1721382065.816663 mapwrite 0x4c4082 thru 0x4d0ec6 (0xce45 bytes) truncating to largest ever: 0x1120c2 000017[1] 1721382065.846729 trunc from 0x4d0ec7 to 0x1120c1 (0x3bee05 bytes) truncating to largest ever: 0x7dd33e 000018[0] 1721382065.855400 trunc from 0x1120c2 to 0x7dd33d (0x6cb27c bytes) 000019[1] 1721382065.864405 trunc from 0x7dd33e to 0x573c68 (0x2696d5 bytes) 000020[0] 1721382065.876719 mapwrite 0x192356 thru 0x194df3 (0x2a9e bytes) 000025[1] 1721382065.916645 trunc from 0x573c69 to 0x51ab84 (0x590e4 bytes) 000028[1] 1721382065.925437 mapread 0x49dbb6 thru 0x49ff86 (0x23d1 bytes) 000029[1] 1721382065.932361 write 0x3a007e thru 0x3a522f (0x51b2 bytes) 000031[1] 1721382065.936943 read 0x20ceb7 thru 0x217869 (0xa9b3 bytes) 000036[1] 1721382065.952083 mapread 0x428775 thru 0x429a28 (0x12b4 bytes) 000038[1] 1721382065.963209 mapwrite 0x9f06ed thru 0x9fa128 (0x9a3c bytes) 000039[0] 1721382065.994289 mapwrite 0x55c48f thru 0x569acf (0xd641 bytes) truncating to largest ever: 0x817d5d 000044[0] 1721382066.035966 trunc from 0x9fa129 to 0x817d5c (0x1e23cc bytes) 000045[1] 1721382066.044489 mapwrite 0x0fec70 thru 0x1082b8 (0x9649 bytes) 000046[0] 1721382066.076265 trunc from 0x817d5d to 0x4df397 (0x3389c5 bytes) 000047[1] 1721382066.084859 read 0x19284c thru 0x19ffd5 (0xd78a bytes) 000049[1] 1721382066.097615 trunc from 0x4df398 to 0x3db167 (0x104230 bytes) 000050[0] 1721382066.107047 read 0x1e0b62 thru 0x1e3904 (0x2da3 bytes) 000052[1] 1721382066.114485 write 0x7f72cb thru 0x7f740f (0x145 bytes) 000054[1] 1721382066.117503 mapread 0x2b9925 thru 0x2c2640 (0x8d1c bytes) 000055[0] 1721382066.134053 write 0x09ea0c thru 0x0aa7b1 (0xbda6 bytes) 000056[1] 1721382066.143137 write 0x30c2b7 thru 0x314be1 (0x892b bytes) 000057[1] 1721382066.146204 mapwrite 0x02bf97 thru 0x033558 (0x75c2 bytes) 000058[0] 1721382066.171124 trunc from 0x7f7410 to 0x0b2dc9 (0x744646 bytes) truncating to largest ever: 0x8c321c 000060[1] 1721382066.186362 trunc from 0x0b2dca to 0x8c321b (0x810452 bytes) 000061[1] 1721382066.202215 mapread 0x45ae2b thru 0x4647cd (0x99a3 bytes) 000062[0] 1721382066.218225 mapread 0x72a2cf thru 0x72f4a4 (0x51d6 bytes) 000063[1] 1721382066.227998 trunc from 0x8c321c to 0x107037 (0x7bc1e4 bytes) truncating to largest ever: 0x8cf3cf 000064[0] 1721382066.234560 trunc from 0x107038 to 0x8cf3ce (0x7c8397 bytes) 000066[0] 1721382066.247131 mapread 0x3a2040 thru 0x3ad74b (0xb70c bytes) 000067[1] 1721382066.256980 write 0x510168 thru 0x5198c1 (0x975a bytes) 000069[0] 1721382066.266248 trunc from 0x8cf3cf to 0x4fe5bd (0x3d0e11 bytes) 000070[1] 1721382066.278719 mapread 0x0e5094 thru 0x0e7bbd (0x2b2a bytes) 000071[1] 1721382066.288681 mapread 0x311fe9 thru 0x31b641 (0x9659 bytes) 000072[0] 1721382066.306245 trunc from 0x4fe5be to 0x5ebcc6 (0xed709 bytes) 000073[0] 1721382066.316508 read 0x427790 thru 0x433b3e (0xc3af bytes) 000074[1] 1721382066.328219 mapread 0x12b643 thru 0x137738 (0xc0f6 bytes) 000075[1] 1721382066.348568 mapread 0x338a6b thru 0x344b24 (0xc0ba bytes) 000076[1] 1721382066.365853 mapread 0x2c7c3c thru 0x2cf0e9 (0x74ae bytes) 000077[1] 1721382066.380122 mapwrite 0x8373ce thru 0x83dba6 (0x67d9 bytes) 000078[1] 1721382066.410434 read 0x75349c thru 0x7623a4 (0xef09 bytes) 000080[1] 1721382066.419156 trunc from 0x83dba7 to 0x60cbfb (0x230fab bytes) 000081[0] 1721382066.426756 write 0x2a8ca1 thru 0x2a9037 (0x397 bytes) 000083[1] 1721382066.436605 trunc from 0x60cbfc to 0x7c168f (0x1b4a94 bytes) 000084[1] 1721382066.443668 trunc from 0x7c1690 to 0x083d55 (0x73d93a bytes) 000085[0] 1721382066.450864 trunc from 0x083d56 to 0x18695e (0x102c09 bytes) 000090[0] 1721382066.468620 write 0x039497 thru 0x03c602 (0x316c bytes) 000091[0] 1721382066.473767 mapwrite 0x672f82 thru 0x67f446 (0xc4c5 bytes) 000093[0] 1721382066.505170 trunc from 0x67f447 to 0x23e43b (0x44100b bytes) 000097[1] 1721382066.515138 mapread 0x163027 thru 0x16622c (0x3206 bytes) 000099[0] 1721382066.525875 mapwrite 0x510c3d thru 0x51f5f4 (0xe9b8 bytes) 000100[1] 1721382066.565603 mapwrite 0x065477 thru 0x0662b1 (0xe3b bytes) 000104[1] 1721382066.597826 write 0x9e6ec8 thru 0x9ecc1d (0x5d56 bytes) 000105[1] 1721382066.606002 mapread 0x33e546 thru 0x34787e (0x9339 bytes) 000106[1] 1721382066.625851 mapwrite 0x26a4b8 thru 0x270c91 (0x67da bytes) 000107[1] 1721382066.659103 mapread 0x16a122 thru 0x1755ef (0xb4ce bytes) 000109[0] 1721382066.682263 read 0x4ec594 thru 0x4f79aa (0xb417 bytes) 000111[1] 1721382066.688351 trunc from 0x9ecc1e to 0x1f4fd2 (0x7f7c4b bytes) 000112[1] 1721382066.698449 read 0x1ef8a5 thru 0x1f0a6b (0x11c7 bytes) 000113[1] 1721382066.705134 trunc from 0x1f4fd3 to 0x028388 (0x1ccc4a bytes) 000115[1] 1721382066.715374 mapwrite 0x7ea68b thru 0x7f3169 (0x8adf bytes) 000117[0] 1721382066.755513 trunc from 0x7f316a to 0x4f2b43 (0x300626 bytes) 000118[0] 1721382066.768218 write 0x45fd83 thru 0x4669c0 (0x6c3e bytes) 000119[0] 1721382066.774023 mapread 0x3dfead thru 0x3e016b (0x2bf bytes) 000121[0] 1721382066.776321 write 0x1cdadd thru 0x1d6b25 (0x9049 bytes) 000122[0] 1721382066.778870 read 0x4d91b5 thru 0x4dcc5f (0x3aab bytes) 000124[0] 1721382066.787394 mapwrite 0x692870 thru 0x69938a (0x6b1b bytes) 000125[1] 1721382066.816649 read 0x0d4f6e thru 0x0d7629 (0x26bc bytes) 000126[1] 1721382066.827061 mapwrite 0x9278cd thru 0x92a1dc (0x2910 bytes) 000127[1] 1721382066.881154 mapwrite 0x4b0f68 thru 0x4bb72e (0xa7c7 bytes) 000128[0] 1721382066.895428 trunc from 0x92a1dd to 0x3b43ff (0x575ddd bytes) 000129[1] 1721382066.913412 trunc from 0x3b4400 to 0x417da3 (0x639a4 bytes) 000130[1] 1721382066.922396 mapwrite 0x1a4041 thru 0x1b10b8 (0xd078 bytes) 000131[0] 1721382066.957085 mapread 0x0db111 thru 0x0e981c (0xe70c bytes) 000132[0] 1721382066.978037 trunc from 0x417da4 to 0x6e15af (0x2c980c bytes) 000133[0] 1721382066.990084 mapread 0x3512d7 thru 0x35983a (0x8564 bytes) 000134[1] 1721382067.003672 trunc from 0x6e15b0 to 0x77d2f4 (0x9bd45 bytes) 000135[0] 1721382067.010736 write 0x3464d5 thru 0x34a24e (0x3d7a bytes) 000140[0] 1721382067.031105 read 0x40cd7b thru 0x4139c6 (0x6c4c bytes) 000141[0] 1721382067.047886 trunc from 0x77d2f5 to 0x57db5a (0x1ff79a bytes) 000142[1] 1721382067.063023 mapread 0x38bb66 thru 0x38d159 (0x15f4 bytes) 000143[1] 1721382067.075532 write 0x327791 thru 0x333b6b (0xc3db bytes) 000144[1] 1721382067.082698 trunc from 0x57db5b to 0x5b1edc (0x34382 bytes) 000146[1] 1721382067.094466 read 0x330cd4 thru 0x33ec4c (0xdf79 bytes) 000151[1] 1721382067.115258 write 0x4fd406 thru 0x503fef (0x6bea bytes) 000155[1] 1721382067.119347 read 0x3fa4d9 thru 0x407f21 (0xda49 bytes) 000157[1] 1721382067.126056 read 0x397864 thru 0x3a01cc (0x8969 bytes) 000159[0] 1721382067.130122 trunc from 0x5b1edd to 0x603908 (0x51a2c bytes) 000163[1] 1721382067.157157 mapwrite 0x1aeafb thru 0x1b2b05 (0x400b bytes) 000166[0] 1721382067.190356 write 0x396c33 thru 0x39c7f1 (0x5bbf bytes) 000167[1] 1721382067.198944 trunc from 0x603909 to 0x329579 (0x2da38f bytes) 000168[1] 1721382067.212454 mapread 0x1e5ca2 thru 0x1e8cf2 (0x3051 bytes) truncating to largest ever: 0x9846ac 000170[0] 1721382067.222132 trunc from 0x32957a to 0x9846ab (0x65b132 bytes) 000171[1] 1721382067.231450 read 0x3ef029 thru 0x3f2829 (0x3801 bytes) 000172[0] 1721382067.237176 mapwrite 0x87732d thru 0x87a3d2 (0x30a6 bytes) 000173[1] 1721382067.270261 trunc from 0x9846ac to 0x7f2eb9 (0x1917f2 bytes) 000177[0] 1721382067.293423 mapread 0x287d4c thru 0x296185 (0xe43a bytes) 000181[1] 1721382067.324668 mapwrite 0x2218ce thru 0x2237da (0x1f0d bytes) 000183[1] 1721382067.359418 mapwrite 0x385589 thru 0x388f35 (0x39ad bytes) 000184[1] 1721382067.387610 read 0x3d541d thru 0x3ded57 (0x993b bytes) 000186[0] 1721382067.392690 read 0x78f228 thru 0x795d78 (0x6b51 bytes) 000187[0] 1721382067.405080 mapread 0x03dbf7 thru 0x04d3d6 (0xf7e0 bytes) 000188[0] 1721382067.452898 mapwrite 0x8f7f0d thru 0x901964 (0x9a58 bytes) 000191[0] 1721382067.493714 mapread 0x127dd8 thru 0x137235 (0xf45e bytes) 000192[0] 1721382067.524887 mapread 0x7ce14c thru 0x7db57e (0xd433 bytes) 000193[0] 1721382067.544713 read 0x72d234 thru 0x73990a (0xc6d7 bytes) 000195[0] 1721382067.549783 mapwrite 0x87966a thru 0x88090c (0x72a3 bytes) 000196[0] 1721382067.567245 write 0x7e80a8 thru 0x7efef5 (0x7e4e bytes) 000198[1] 1721382067.570913 mapwrite 0x15587e thru 0x1654d2 (0xfc55 bytes) 000199[0] 1721382067.628141 trunc from 0x901965 to 0x8a7265 (0x5a6ff bytes) 000200[0] 1721382067.637199 read 0x083839 thru 0x0897db (0x5fa3 bytes) 000201[1] 1721382067.656431 mapread 0x134ca6 thru 0x13d155 (0x84b0 bytes) 000202[0] 1721382067.679927 read 0x6a239c thru 0x6b0f4d (0xebb2 bytes) 000203[1] 1721382067.681028 trunc from 0x8a7266 to 0x30a39e (0x59cec7 bytes) 000204[1] 1721382067.693173 mapwrite 0x20522b thru 0x20eb82 (0x9958 bytes) 000205[1] 1721382067.726647 write 0x95692a thru 0x958fe7 (0x26be bytes) 000207[1] 1721382067.735682 write 0x39dff1 thru 0x3a2b5a (0x4b6a bytes) 000208[1] 1721382067.748081 trunc from 0x958fe8 to 0x169549 (0x7efa9e bytes) 000209[1] 1721382067.759927 mapread 0x01446b thru 0x020c93 (0xc829 bytes) 000210[1] 1721382067.777369 write 0x786510 thru 0x78a330 (0x3e21 bytes) 000212[1] 1721382067.785056 write 0x3cff36 thru 0x3da39a (0xa465 bytes) 000213[0] 1721382067.790479 mapread 0x5cd7ce thru 0x5d4f6f (0x77a2 bytes) 000214[1] 1721382067.808990 mapwrite 0x20364f thru 0x2089ae (0x5360 bytes) 000215[0] 1721382067.834805 write 0x576fe9 thru 0x57ce54 (0x5e6c bytes) 000216[0] 1721382067.842804 mapread 0x0dbcb5 thru 0x0e31c2 (0x750e bytes) 000218[0] 1721382067.862123 mapwrite 0x5647c8 thru 0x56cd97 (0x85d0 bytes) 000220[0] 1721382067.890313 trunc from 0x78a331 to 0x630d47 (0x1595e9 bytes) 000223[1] 1721382067.904010 write 0x0e9b4b thru 0x0f2e78 (0x932e bytes) 000227[1] 1721382067.924376 write 0x6839bd thru 0x6922da (0xe91e bytes) 000228[1] 1721382067.930079 trunc from 0x6922db to 0x4522e5 (0x23fff5 bytes) 000229[0] 1721382067.941541 read 0x391181 thru 0x397498 (0x6318 bytes) 000231[0] 1721382067.955789 trunc from 0x4522e6 to 0x31cb5d (0x135788 bytes) 000234[0] 1721382067.978351 trunc from 0x31cb5e to 0x03e706 (0x2de457 bytes) 000237[0] 1721382067.986035 trunc from 0x03e707 to 0x5be0a4 (0x57f99e bytes) 000238[0] 1721382067.991717 write 0x70ab79 thru 0x710274 (0x56fc bytes) 000242[1] 1721382068.005236 trunc from 0x710275 to 0x0b2190 (0x65e0e4 bytes) 000244[1] 1721382068.019330 mapwrite 0x9bc48f thru 0x9c3755 (0x72c7 bytes) 000245[1] 1721382068.048256 write 0x85f55e thru 0x865563 (0x6006 bytes) 000246[0] 1721382068.056980 mapwrite 0x2a81a3 thru 0x2b2356 (0xa1b4 bytes) 000249[0] 1721382068.104845 mapread 0x84e6a4 thru 0x8508fe (0x225b bytes) 000250[1] 1721382068.112609 trunc from 0x9c3756 to 0x5b511c (0x40e639 bytes) 000251[1] 1721382068.123146 read 0x16dda1 thru 0x178a3e (0xac9e bytes) 000254[0] 1721382068.152373 read 0x3a8ba7 thru 0x3b3562 (0xa9bc bytes) 000255[1] 1721382068.168827 write 0x8bc80a thru 0x8c8ce0 (0xc4d7 bytes) 000256[0] 1721382068.172682 mapread 0x17c0cb thru 0x17d8d8 (0x180e bytes) 000257[0] 1721382068.177645 read 0x70d210 thru 0x70d757 (0x548 bytes) 000258[1] 1721382068.181937 write 0x64cbd9 thru 0x654745 (0x7b6d bytes) 000259[0] 1721382068.192352 mapwrite 0x201cd2 thru 0x2058dc (0x3c0b bytes) 000261[1] 1721382068.232277 trunc from 0x8c8ce1 to 0x8be8fe (0xa3e2 bytes) 000262[0] 1721382068.241778 trunc from 0x8be8ff to 0x28ed90 (0x62fb6e bytes) 000264[1] 1721382068.255591 mapwrite 0x8f2fcf thru 0x9014ec (0xe51e bytes) 000273[1] 1721382068.293693 mapread 0x708ca6 thru 0x71383b (0xab96 bytes) 000274[1] 1721382068.312382 trunc from 0x9014ed to 0x941903 (0x40417 bytes) 000275[1] 1721382068.321062 write 0x896644 thru 0x899152 (0x2b0f bytes) 000277[0] 1721382068.325763 trunc from 0x941904 to 0x39eb5e (0x5a2da5 bytes) 000281[0] 1721382068.346486 mapread 0x357943 thru 0x359fef (0x26ad bytes) 000282[0] 1721382068.353190 mapread 0x056571 thru 0x05b58b (0x501b bytes) 000283[0] 1721382068.359449 trunc from 0x39eb5f to 0x8cc53f (0x52d9e1 bytes) 000285[0] 1721382068.371683 mapread 0x885558 thru 0x88d931 (0x83da bytes) 000288[1] 1721382068.383625 read 0x5a2ee5 thru 0x5a4132 (0x124e bytes) 000291[0] 1721382068.393889 trunc from 0x8cc540 to 0x674ebd (0x257682 bytes) truncating to largest ever: 0x9b2cf5 000292[0] 1721382068.403853 trunc from 0x674ebe to 0x9b2cf4 (0x33de37 bytes) 000293[0] 1721382068.412968 write 0x4b666a thru 0x4bb686 (0x501d bytes) 000296[0] 1721382068.426550 read 0x11f6cd thru 0x12c3c8 (0xccfc bytes) 000297[0] 1721382068.447163 trunc from 0x9b2cf5 to 0x79daf9 (0x2151fb bytes) 000298[0] 1721382068.453306 write 0x62f851 thru 0x63313a (0x38ea bytes) 000299[0] 1721382068.453700 trunc from 0x79dafa to 0x9a1443 (0x20394a bytes) 000301[0] 1721382068.461263 read 0x04cf66 thru 0x05c01c (0xf0b7 bytes) 000302[1] 1721382068.473534 write 0x35a6de thru 0x35bbe4 (0x1507 bytes) 000303[0] 1721382068.484694 read 0x034418 thru 0x043310 (0xeef9 bytes) 000305[0] 1721382068.493153 read 0x2b9333 thru 0x2c350d (0xa1db bytes) 000306[0] 1721382068.498636 trunc from 0x9a1444 to 0x1ad9d8 (0x7f3a6b bytes) 000307[0] 1721382068.515305 write 0x7d78dd thru 0x7da09e (0x27c2 bytes) 000308[0] 1721382068.518638 mapread 0x588361 thru 0x5955ad (0xd24d bytes) 000309[1] 1721382068.536026 trunc from 0x7da09f to 0x2089cc (0x5d16d2 bytes) 000310[1] 1721382068.545295 trunc from 0x2089cd to 0x188a4f (0x7ff7d bytes) 000312[0] 1721382068.548789 trunc from 0x188a50 to 0x8073c0 (0x67e971 bytes) 000316[0] 1721382068.566225 mapwrite 0x3268a4 thru 0x332942 (0xc09f bytes) 000318[0] 1721382068.595464 mapread 0x5fcfa0 thru 0x600023 (0x3084 bytes) 000322[0] 1721382068.606425 trunc from 0x8073c1 to 0x195d83 (0x67163d bytes) 000323[0] 1721382068.616649 write 0x368fc5 thru 0x36d0db (0x4117 bytes) 000324[0] 1721382068.618940 mapwrite 0x294c49 thru 0x2a426b (0xf623 bytes) 000325[0] 1721382068.650933 write 0x1fa976 thru 0x2061bd (0xb848 bytes) 000326[1] 1721382068.658138 write 0x88080d thru 0x883351 (0x2b45 bytes) 000328[0] 1721382068.670483 trunc from 0x883352 to 0x7fb500 (0x87e51 bytes) 000330[0] 1721382068.681472 write 0x156a5a thru 0x1608a6 (0x9e4d bytes) 000331[1] 1721382068.687290 read 0x5a220e thru 0x5a7b13 (0x5906 bytes) 000332[0] 1721382068.695300 write 0x80e37b thru 0x80f1d8 (0xe5e bytes) 000334[1] 1721382068.695824 write 0x8647ba thru 0x86d03c (0x8883 bytes) 000336[0] 1721382068.702645 read 0x680a1b thru 0x688f35 (0x851b bytes) 000339[0] 1721382068.717854 write 0x25bdb8 thru 0x260a15 (0x4c5e bytes) 000340[1] 1721382068.725167 mapwrite 0x209e83 thru 0x211566 (0x76e4 bytes) 000341[0] 1721382068.760795 mapwrite 0x20a44e thru 0x210d72 (0x6925 bytes) 000342[1] 1721382068.782895 trunc from 0x86d03d to 0x5bf23b (0x2ade01 bytes) 000343[0] 1721382068.790825 write 0x35d655 thru 0x36b7b9 (0xe165 bytes) 000344[1] 1721382068.797365 mapread 0x3eb0df thru 0x3f9e3f (0xed61 bytes) 000346[0] 1721382068.819538 mapwrite 0x72027e thru 0x7295b0 (0x9333 bytes) 000350[0] 1721382068.861131 trunc from 0x7295b1 to 0x39e686 (0x38af2a bytes) 000351[1] 1721382068.867624 trunc from 0x39e687 to 0x5467d0 (0x1a814a bytes) 000353[1] 1721382068.877510 read 0x3feb5c thru 0x40653e (0x79e3 bytes) 000354[0] 1721382068.888000 mapread 0x070215 thru 0x074c4d (0x4a39 bytes) 000356[0] 1721382068.903226 mapread 0x539150 thru 0x5467d0 (0xd681 bytes) 000357[0] 1721382068.920465 write 0x599aa6 thru 0x59c045 (0x25a0 bytes) 000359[0] 1721382068.926633 read 0x2831af thru 0x28f720 (0xc572 bytes) 000360[1] 1721382068.938325 write 0x8ac09d thru 0x8aee84 (0x2de8 bytes) 000363[1] 1721382068.950993 mapread 0x201408 thru 0x20c4c4 (0xb0bd bytes) 000365[0] 1721382068.970218 trunc from 0x8aee85 to 0x692d8b (0x21c0f9 bytes) 000366[1] 1721382068.977093 mapwrite 0x142be5 thru 0x1443d2 (0x17ee bytes) 000367[0] 1721382069.008645 read 0x4092ff thru 0x40d6ca (0x43cc bytes) 000368[1] 1721382069.018702 mapwrite 0x6f046a thru 0x6f583d (0x53d4 bytes) 000369[1] 1721382069.046293 trunc from 0x6f583e to 0x8954f0 (0x19fcb3 bytes) 000372[0] 1721382069.053995 mapwrite 0x0f1d11 thru 0x0f52af (0x359f bytes) 000375[0] 1721382069.084805 read 0x5d2513 thru 0x5d33d7 (0xec5 bytes) 000378[0] 1721382069.101700 write 0x7c9c39 thru 0x7cf75d (0x5b25 bytes) 000380[1] 1721382069.112835 read 0x34490a thru 0x34cf13 (0x860a bytes) 000382[1] 1721382069.123091 write 0x731295 thru 0x7410cb (0xfe37 bytes) 000385[0] 1721382069.139404 write 0x22ada9 thru 0x22ff7d (0x51d5 bytes) 000386[0] 1721382069.144349 write 0x8952c2 thru 0x8970bc (0x1dfb bytes) 000387[1] 1721382069.147390 mapwrite 0x36bc4e thru 0x374170 (0x8523 bytes) 000389[0] 1721382069.177995 mapread 0x738ee7 thru 0x73c3fa (0x3514 bytes) 000390[1] 1721382069.188765 mapwrite 0x8eef7f thru 0x8f7ad9 (0x8b5b bytes) 000391[0] 1721382069.228783 mapwrite 0x3bf0c2 thru 0x3c7b6a (0x8aa9 bytes) 000394[1] 1721382069.269444 read 0x711c39 thru 0x71f001 (0xd3c9 bytes) 000395[1] 1721382069.277035 mapwrite 0x9acdd4 thru 0x9b7060 (0xa28d bytes) 000397[1] 1721382069.308628 mapread 0x7a9ada thru 0x7ac61f (0x2b46 bytes) 000398[1] 1721382069.315539 write 0x92c84e thru 0x9303c5 (0x3b78 bytes) 000399[1] 1721382069.316174 mapwrite 0x8b1a63 thru 0x8b2a3b (0xfd9 bytes) 000401[1] 1721382069.328496 mapwrite 0x17dcdc thru 0x182704 (0x4a29 bytes) 000402[1] 1721382069.350358 mapwrite 0x67dcc1 thru 0x688bfc (0xaf3c bytes) 000403[0] 1721382069.368643 mapread 0x4937f1 thru 0x49bdb2 (0x85c2 bytes) 000404[1] 1721382069.385439 write 0x32e0ce thru 0x33727c (0x91af bytes) 000405[0] 1721382069.390418 trunc from 0x9b7061 to 0x136698 (0x8809c8 bytes) 000407[0] 1721382069.399941 mapread 0x0a0a07 thru 0x0a7e75 (0x746f bytes) 000410[0] 1721382069.412264 read 0x0ebd51 thru 0x0ed33c (0x15ec bytes) 000411[0] 1721382069.415100 mapread 0x06a3db thru 0x072e61 (0x8a87 bytes) 000413[1] 1721382069.427232 write 0x4e2aad thru 0x4ecb93 (0xa0e7 bytes) 000414[1] 1721382069.433069 mapread 0x37a126 thru 0x37ec54 (0x4b2f bytes) 000416[1] 1721382069.447542 read 0x064427 thru 0x06a1a8 (0x5d82 bytes) 000417[1] 1721382069.459242 mapwrite 0x9b9cc6 thru 0x9bf69d (0x59d8 bytes) 000418[1] 1721382069.496214 mapread 0x36b3ef thru 0x378f0a (0xdb1c bytes) 000419[0] 1721382069.521247 mapread 0x6f8553 thru 0x6fcd27 (0x47d5 bytes) 000422[1] 1721382069.546256 mapwrite 0x0edac2 thru 0x0fce72 (0xf3b1 bytes) 000424[0] 1721382069.582123 mapread 0x44644f thru 0x448ffc (0x2bae bytes) 000426[0] 1721382069.591438 mapread 0x1f02a2 thru 0x1f59e9 (0x5748 bytes) 000427[0] 1721382069.615887 write 0x17f0ee thru 0x18ee27 (0xfd3a bytes) 000432[0] 1721382069.638481 write 0x273a0f thru 0x274b2b (0x111d bytes) 000433[1] 1721382069.649585 trunc from 0x9bf69e to 0x935206 (0x8a497 bytes) 000434[1] 1721382069.666130 mapwrite 0x60f919 thru 0x61b6bb (0xbda3 bytes) 000436[0] 1721382069.696492 trunc from 0x935207 to 0x4bc2bf (0x478f47 bytes) 000437[0] 1721382069.703901 read 0x2c5bdc thru 0x2d183b (0xbc60 bytes) 000443[1] 1721382069.717378 write 0x8e4a8b thru 0x8f3fc7 (0xf53d bytes) 000447[1] 1721382069.733205 read 0x0d3f99 thru 0x0e08e5 (0xc94d bytes) 000450[1] 1721382069.745209 write 0x5bcc4f thru 0x5cc447 (0xf7f9 bytes) 000451[0] 1721382069.769317 mapwrite 0x3961f0 thru 0x39f1c7 (0x8fd8 bytes) 000452[0] 1721382069.808086 mapread 0x74eb74 thru 0x752436 (0x38c3 bytes) 000453[1] 1721382069.824589 mapread 0x7cd53d thru 0x7d4cd5 (0x7799 bytes) 000454[1] 1721382069.838533 mapwrite 0x065b51 thru 0x072804 (0xccb4 bytes) 000458[1] 1721382069.880456 mapwrite 0x565113 thru 0x56af9a (0x5e88 bytes) 000459[1] 1721382069.901331 write 0x972a1e thru 0x97543d (0x2a20 bytes) 000460[0] 1721382069.901861 trunc from 0x97543e to 0x75ad66 (0x21a6d7 bytes) 000461[0] 1721382069.919145 mapread 0x6b267b thru 0x6b3421 (0xda7 bytes) 000467[0] 1721382069.943163 write 0x6d6a52 thru 0x6d848e (0x1a3d bytes) 000469[0] 1721382069.947193 trunc from 0x75ad67 to 0x6d236b (0x889fb bytes) 000471[1] 1721382069.958134 mapwrite 0x6acf03 thru 0x6babab (0xdca9 bytes) 000472[0] 1721382069.992244 write 0x9b02fb thru 0x9b1c6f (0x1975 bytes) 000473[1] 1721382069.996394 write 0x025525 thru 0x028544 (0x3020 bytes) 000475[0] 1721382070.003257 mapwrite 0x3c68c3 thru 0x3d03b1 (0x9aef bytes) 000478[0] 1721382070.042581 mapread 0x0a3f10 thru 0x0a659f (0x2690 bytes) 000479[0] 1721382070.053658 mapwrite 0x66ebe3 thru 0x67cc36 (0xe054 bytes) 000481[0] 1721382070.089908 read 0x3cfa63 thru 0x3d8d30 (0x92ce bytes) 000483[1] 1721382070.100853 write 0x476574 thru 0x480960 (0xa3ed bytes) 000485[0] 1721382070.118879 write 0x578e0a thru 0x58164c (0x8843 bytes) 000486[0] 1721382070.122430 mapwrite 0x615806 thru 0x618ec3 (0x36be bytes) 000489[1] 1721382070.150240 trunc from 0x9b1c70 to 0x7f04f3 (0x1c177c bytes) 000490[0] 1721382070.164213 write 0x575489 thru 0x57b523 (0x609b bytes) 000491[0] 1721382070.173211 mapwrite 0x64dd82 thru 0x65018e (0x240d bytes) 000493[0] 1721382070.191717 mapwrite 0x54317f thru 0x54a1c4 (0x7046 bytes) 000494[1] 1721382070.218492 write 0x81370a thru 0x821da1 (0xe698 bytes) 000495[1] 1721382070.226034 mapwrite 0x6461df thru 0x64e47a (0x829c bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2070 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1721382070.457889 mapwrite 0x77c000 thru 0x783e6e (0x7e6f bytes) 000005[0] 1721382070.494175 mapread 0x6f5000 thru 0x703494 (0xe495 bytes) 000009[1] 1721382070.504869 read_OD 0x2be000 thru 0x2c8fff (0xb000 bytes) 000010[0] 1721382070.507039 write 0x19c000 thru 0x1aa5de (0xe5df bytes) 000012[1] 1721382070.510514 write_OD 0x28f000 thru 0x298fff (0xa000 bytes) truncating to largest ever: 0x93f858 000016[0] 1721382070.528596 trunc from 0x783e6f to 0x93f857 (0x1bb9e9 bytes) 000018[0] 1721382070.537195 mapwrite 0x913000 thru 0x922984 (0xf985 bytes) 000022[0] 1721382070.562521 read 0x72a000 thru 0x733b99 (0x9b9a bytes) 000024[0] 1721382070.567582 trunc from 0x93f858 to 0x74f79f (0x1f00b8 bytes) 000026[0] 1721382070.575278 mapwrite 0x8ea000 thru 0x8ebcab (0x1cac bytes) 000027[1] 1721382070.593057 trunc from 0x8ebcac to 0x8e0f60 (0xad4b bytes) 000028[0] 1721382070.600913 trunc from 0x8e0f61 to 0x916f01 (0x35fa1 bytes) 000029[1] 1721382070.608050 read_OD 0x77e000 thru 0x783fff (0x6000 bytes) 000030[1] 1721382070.612965 mapread 0x133000 thru 0x141e29 (0xee2a bytes) 000033[0] 1721382070.632729 mapwrite 0x9aa000 thru 0x9b6d68 (0xcd69 bytes) 000035[1] 1721382070.660223 trunc from 0x9b6d69 to 0x249096 (0x76dcd2 bytes) 000036[1] 1721382070.666364 read_OD 0x224000 thru 0x227fff (0x4000 bytes) 000039[1] 1721382070.671402 trunc from 0x249097 to 0x754c94 (0x50bbfe bytes) 000041[1] 1721382070.682824 write_OD 0x900000 thru 0x90dfff (0xe000 bytes) 000044[1] 1721382070.698625 trunc from 0x90e000 to 0x4d3457 (0x43aba8 bytes) 000046[1] 1721382070.710868 trunc from 0x4d3458 to 0x3576b4 (0x17bda3 bytes) 000047[1] 1721382070.715046 trunc from 0x3576b5 to 0x1d79ba (0x17fcfa bytes) 000055[0] 1721382070.730618 mapread 0x139000 thru 0x14026e (0x726f bytes) 000056[0] 1721382070.746041 read 0x028000 thru 0x028b18 (0xb19 bytes) 000057[0] 1721382070.754070 mapread 0x1bb000 thru 0x1c3da2 (0x8da3 bytes) 000058[0] 1721382070.754441 trunc from 0x1d79bb to 0x766aca (0x58f110 bytes) 000059[1] 1721382070.766778 mapwrite 0x0c9000 thru 0x0d18c1 (0x88c2 bytes) 000060[0] 1721382070.807326 read 0x590000 thru 0x59c428 (0xc429 bytes) 000062[0] 1721382070.822691 read 0x486000 thru 0x491b77 (0xbb78 bytes) 000067[0] 1721382070.830282 read 0x23b000 thru 0x2406bb (0x56bc bytes) 000069[0] 1721382070.841789 mapread 0x39b000 thru 0x3a53a3 (0xa3a4 bytes) 000072[1] 1721382070.842684 write_OD 0x400000 thru 0x409fff (0xa000 bytes) 000073[1] 1721382070.862304 write_OD 0x9fb000 thru 0x9fbfff (0x1000 bytes) 000074[1] 1721382070.878411 mapwrite 0x048000 thru 0x054384 (0xc385 bytes) 000076[0] 1721382070.939196 mapread 0x629000 thru 0x62e5a1 (0x55a2 bytes) 000079[1] 1721382070.953153 mapread 0x5ae000 thru 0x5b9a7d (0xba7e bytes) 000080[0] 1721382070.964785 mapread 0x088000 thru 0x0891c0 (0x11c1 bytes) 000081[0] 1721382070.967313 read_OD 0x32f000 thru 0x331fff (0x3000 bytes) 000082[1] 1721382070.968652 mapwrite 0x6ab000 thru 0x6ad02c (0x202d bytes) 000083[1] 1721382070.990621 mapread 0x7ef000 thru 0x7fef7b (0xff7c bytes) 000087[1] 1721382071.011747 write_OD 0x294000 thru 0x297fff (0x4000 bytes) 000089[1] 1721382071.025585 trunc from 0x9fc000 to 0x1720ad (0x889f52 bytes) 000090[0] 1721382071.030817 write_OD 0x23c000 thru 0x240fff (0x5000 bytes) 000091[0] 1721382071.049437 write_OD 0x4b0000 thru 0x4bbfff (0xc000 bytes) 000095[1] 1721382071.065127 write_OD 0x7c5000 thru 0x7cbfff (0x7000 bytes) 000096[1] 1721382071.082856 write_OD 0x0af000 thru 0x0b6fff (0x8000 bytes) 000098[0] 1721382071.096250 trunc from 0x7cc000 to 0x55a68d (0x271972 bytes) 000099[0] 1721382071.105107 trunc from 0x55a68e to 0x09817e (0x4c250f bytes) 000101[1] 1721382071.108823 write_OD 0x01c000 thru 0x026fff (0xb000 bytes) 000102[1] 1721382071.122827 read_OD 0x00d000 thru 0x017fff (0xb000 bytes) 000103[1] 1721382071.125606 mapwrite 0x37d000 thru 0x383a11 (0x6a12 bytes) 000104[0] 1721382071.162889 mapwrite 0x672000 thru 0x681f58 (0xff59 bytes) 000105[1] 1721382071.194424 write_OD 0x959000 thru 0x95bfff (0x3000 bytes) 000107[0] 1721382071.209522 mapwrite 0x7a2000 thru 0x7b04c6 (0xe4c7 bytes) 000109[1] 1721382071.236372 read_OD 0x4a3000 thru 0x4a7fff (0x5000 bytes) 000111[1] 1721382071.243158 write_OD 0x4a3000 thru 0x4a3fff (0x1000 bytes) 000112[0] 1721382071.256709 mapwrite 0x72d000 thru 0x73cbd6 (0xfbd7 bytes) 000115[0] 1721382071.282344 mapwrite 0x218000 thru 0x222cfa (0xacfb bytes) 000116[1] 1721382071.306084 trunc from 0x95c000 to 0x1f50b0 (0x766f4f bytes) 000117[0] 1721382071.314896 read_OD 0x041000 thru 0x04bfff (0xb000 bytes) 000118[1] 1721382071.319482 mapwrite 0x6f8000 thru 0x6f9a71 (0x1a72 bytes) 000119[0] 1721382071.353843 write_OD 0x24f000 thru 0x25bfff (0xd000 bytes) 000120[0] 1721382071.370523 read_OD 0x1a4000 thru 0x1aefff (0xb000 bytes) 000122[1] 1721382071.374213 mapwrite 0x322000 thru 0x327922 (0x5923 bytes) 000124[1] 1721382071.398688 mapread 0x57e000 thru 0x580004 (0x2005 bytes) 000126[1] 1721382071.405084 read_OD 0x283000 thru 0x28afff (0x8000 bytes) 000127[1] 1721382071.409384 write_OD 0x74f000 thru 0x759fff (0xb000 bytes) truncating to largest ever: 0x9e0c52 000128[1] 1721382071.427150 trunc from 0x75a000 to 0x9e0c51 (0x286c52 bytes) 000129[1] 1721382071.439003 mapread 0x797000 thru 0x7a28fa (0xb8fb bytes) 000132[0] 1721382071.462383 mapread 0x657000 thru 0x65d6ae (0x66af bytes) 000133[1] 1721382071.498741 write_OD 0x57c000 thru 0x57efff (0x3000 bytes) 000134[0] 1721382071.516939 mapread 0x5ba000 thru 0x5ba5bd (0x5be bytes) 000135[1] 1721382071.519342 write_OD 0x7eb000 thru 0x7f5fff (0xb000 bytes) 000136[1] 1721382071.534033 write_OD 0x120000 thru 0x123fff (0x4000 bytes) 000139[0] 1721382071.553462 trunc from 0x9e0c52 to 0x02072c (0x9c0525 bytes) 000141[0] 1721382071.570306 trunc from 0x02072d to 0x2ff5ad (0x2dee81 bytes) 000142[1] 1721382071.576153 read_OD 0x021000 thru 0x028fff (0x8000 bytes) 000145[1] 1721382071.586487 trunc from 0x2ff5ae to 0x8f40b7 (0x5f4b0a bytes) 000146[1] 1721382071.599858 write_OD 0x0fe000 thru 0x100fff (0x3000 bytes) 000148[0] 1721382071.617893 mapread 0x050000 thru 0x057ecd (0x7ece bytes) 000149[0] 1721382071.626256 mapwrite 0x905000 thru 0x908165 (0x3166 bytes) 000151[1] 1721382071.652681 read_OD 0x5f6000 thru 0x5fafff (0x5000 bytes) 000157[1] 1721382071.658355 read_OD 0x28f000 thru 0x297fff (0x9000 bytes) 000159[1] 1721382071.660333 read_OD 0x2f6000 thru 0x301fff (0xc000 bytes) 000160[0] 1721382071.662200 mapread 0x247000 thru 0x252701 (0xb702 bytes) 000162[0] 1721382071.673479 write_OD 0x72b000 thru 0x734fff (0xa000 bytes) 000163[0] 1721382071.686513 read_OD 0x0e1000 thru 0x0e4fff (0x4000 bytes) 000164[1] 1721382071.687672 read_OD 0x3eb000 thru 0x3f6fff (0xc000 bytes) 000165[0] 1721382071.690020 read_OD 0x69a000 thru 0x69cfff (0x3000 bytes) 000166[1] 1721382071.691309 mapread 0x396000 thru 0x39add8 (0x4dd9 bytes) 000170[0] 1721382071.712623 trunc from 0x908166 to 0x2d56db (0x632a8a bytes) 000171[0] 1721382071.725846 write_OD 0x662000 thru 0x66efff (0xd000 bytes) 000172[0] 1721382071.742062 mapwrite 0x90c000 thru 0x915976 (0x9977 bytes) 000176[1] 1721382071.776252 write_OD 0x1a5000 thru 0x1a9fff (0x5000 bytes) 000177[0] 1721382071.791711 mapwrite 0x01f000 thru 0x02899b (0x999c bytes) 000178[0] 1721382071.822214 write_OD 0x751000 thru 0x75afff (0xa000 bytes) 000181[1] 1721382071.835786 mapread 0x674000 thru 0x67b450 (0x7451 bytes) 000186[0] 1721382071.844354 read_OD 0x76b000 thru 0x76cfff (0x2000 bytes) 000190[0] 1721382071.845978 mapread 0x788000 thru 0x78a312 (0x2313 bytes) 000193[0] 1721382071.853730 read_OD 0x138000 thru 0x13afff (0x3000 bytes) 000194[1] 1721382071.855292 read_OD 0x0bd000 thru 0x0c1fff (0x5000 bytes) 000196[1] 1721382071.856968 trunc from 0x915977 to 0x018acd (0x8fcea9 bytes) 000197[1] 1721382071.866845 trunc from 0x018ace to 0x07014b (0x5767e bytes) 000199[1] 1721382071.876838 mapwrite 0x346000 thru 0x347f9d (0x1f9e bytes) 000200[1] 1721382071.904826 mapwrite 0x6a5000 thru 0x6b1a91 (0xca92 bytes) 000201[1] 1721382071.939201 read_OD 0x0e0000 thru 0x0eafff (0xb000 bytes) 000203[1] 1721382071.946994 write_OD 0x12c000 thru 0x134fff (0x9000 bytes) 000204[0] 1721382071.959768 mapwrite 0x456000 thru 0x45ef91 (0x8f92 bytes) 000205[0] 1721382071.995013 read_OD 0x4cd000 thru 0x4cefff (0x2000 bytes) 000206[0] 1721382072.003102 mapwrite 0x596000 thru 0x596dba (0xdbb bytes) 000209[1] 1721382072.031175 trunc from 0x6b1a92 to 0x690d93 (0x20cfe bytes) 000210[0] 1721382072.038188 trunc from 0x690d94 to 0x01aa8c (0x676307 bytes) 000213[0] 1721382072.050551 mapread 0x008000 thru 0x00ef69 (0x6f6a bytes) 000214[1] 1721382072.058203 mapread 0x006000 thru 0x007c52 (0x1c53 bytes) 000216[1] 1721382072.063454 mapwrite 0x2d5000 thru 0x2e0fb9 (0xbfba bytes) 000217[1] 1721382072.095901 write_OD 0x9f6000 thru 0x9fcfff (0x7000 bytes) 000219[1] 1721382072.117040 write_OD 0x297000 thru 0x299fff (0x3000 bytes) 000220[0] 1721382072.130455 trunc from 0x9fd000 to 0x34c5c4 (0x6b0a3b bytes) 000224[0] 1721382072.143784 trunc from 0x34c5c5 to 0x21cec4 (0x12f700 bytes) 000226[1] 1721382072.152737 mapwrite 0x2eb000 thru 0x2eb07d (0x7e bytes) 000228[0] 1721382072.174922 trunc from 0x2eb07e to 0x6fa69e (0x40f621 bytes) 000233[0] 1721382072.187621 write_OD 0x8d4000 thru 0x8ddfff (0xa000 bytes) 000234[1] 1721382072.203559 write_OD 0x70f000 thru 0x714fff (0x6000 bytes) 000235[1] 1721382072.219175 read_OD 0x7a9000 thru 0x7b3fff (0xb000 bytes) 000238[1] 1721382072.222831 read_OD 0x113000 thru 0x11afff (0x8000 bytes) 000240[0] 1721382072.224729 mapread 0x40c000 thru 0x41029d (0x429e bytes) 000243[0] 1721382072.237375 trunc from 0x8de000 to 0x8f28a2 (0x148a3 bytes) 000244[1] 1721382072.250478 trunc from 0x8f28a3 to 0x75dcba (0x194be8 bytes) 000247[0] 1721382072.261382 trunc from 0x75dcbb to 0x7aa4a5 (0x4c7eb bytes) 000249[0] 1721382072.274501 trunc from 0x7aa4a6 to 0x04c94a (0x75db5b bytes) 000250[0] 1721382072.279081 read_OD 0x04b000 thru 0x04bfff (0x1000 bytes) 000252[1] 1721382072.284162 read_OD 0x02b000 thru 0x02efff (0x4000 bytes) 000254[1] 1721382072.285866 trunc from 0x04c94b to 0x90d45b (0x8c0b11 bytes) 000256[1] 1721382072.298502 mapwrite 0x643000 thru 0x643dc0 (0xdc1 bytes) 000259[1] 1721382072.340363 mapread 0x709000 thru 0x709250 (0x251 bytes) 000260[1] 1721382072.343398 mapwrite 0x7bf000 thru 0x7c6798 (0x7799 bytes) 000261[0] 1721382072.359925 trunc from 0x90d45c to 0x7ac99d (0x160abe bytes) 000262[0] 1721382072.373889 read_OD 0x3a4000 thru 0x3abfff (0x8000 bytes) 000267[0] 1721382072.380804 trunc from 0x7ac99e to 0x240545 (0x56c458 bytes) 000268[1] 1721382072.390383 mapread 0x10d000 thru 0x1170f6 (0xa0f7 bytes) 000270[0] 1721382072.406180 read_OD 0x19b000 thru 0x1a8fff (0xe000 bytes) 000272[0] 1721382072.409315 mapread 0x210000 thru 0x218db6 (0x8db7 bytes) 000278[1] 1721382072.415883 mapread 0x1e8000 thru 0x1eb5d5 (0x35d6 bytes) 000279[1] 1721382072.420946 trunc from 0x240546 to 0x688f69 (0x448a24 bytes) 000281[0] 1721382072.434738 trunc from 0x688f6a to 0x0ea0ca (0x59ee9f bytes) 000283[0] 1721382072.439784 trunc from 0x0ea0cb to 0x585276 (0x49b1ac bytes) 000285[1] 1721382072.447353 read_OD 0x4e4000 thru 0x4ebfff (0x8000 bytes) 000286[1] 1721382072.452567 mapread 0x439000 thru 0x43cddb (0x3ddc bytes) 000287[1] 1721382072.460699 trunc from 0x585277 to 0x8be9e5 (0x33976f bytes) 000288[0] 1721382072.467412 write_OD 0x8f5000 thru 0x8fdfff (0x9000 bytes) 000289[0] 1721382072.484087 trunc from 0x8fe000 to 0x90c4ec (0xe4ed bytes) 000290[0] 1721382072.489159 read_OD 0x4b9000 thru 0x4c0fff (0x8000 bytes) 000291[1] 1721382072.492468 trunc from 0x90c4ed to 0x45a7e7 (0x4b1d05 bytes) 000292[0] 1721382072.500444 read_OD 0x27c000 thru 0x282fff (0x7000 bytes) 000293[1] 1721382072.507589 mapwrite 0x7b2000 thru 0x7b87e7 (0x67e8 bytes) 000295[1] 1721382072.543838 write_OD 0x43d000 thru 0x442fff (0x6000 bytes) 000296[0] 1721382072.560211 write_OD 0x4b5000 thru 0x4c0fff (0xc000 bytes) 000297[1] 1721382072.574724 mapread 0x4db000 thru 0x4db85c (0x85d bytes) 000298[0] 1721382072.576223 read_OD 0x441000 thru 0x444fff (0x4000 bytes) 000299[0] 1721382072.578099 write_OD 0x031000 thru 0x03dfff (0xd000 bytes) 000300[1] 1721382072.593584 write_OD 0x6b6000 thru 0x6c4fff (0xf000 bytes) 000301[0] 1721382072.607575 trunc from 0x7b87e8 to 0x7dc1f3 (0x23a0c bytes) 000305[1] 1721382072.616758 trunc from 0x7dc1f4 to 0x0b1570 (0x72ac83 bytes) 000308[0] 1721382072.633215 mapread 0x08e000 thru 0x09222a (0x422b bytes) 000312[1] 1721382072.644090 trunc from 0x0b1571 to 0x0b4cd3 (0x3763 bytes) 000317[1] 1721382072.656596 read 0x080000 thru 0x083a7a (0x3a7b bytes) 000318[1] 1721382072.658236 mapread 0x06e000 thru 0x0719a1 (0x39a2 bytes) 000319[1] 1721382072.665088 mapwrite 0x058000 thru 0x05939d (0x139e bytes) 000322[0] 1721382072.691239 mapread 0x09e000 thru 0x09f7a0 (0x17a1 bytes) 000324[1] 1721382072.698792 trunc from 0x0b4cd4 to 0x118562 (0x6388f bytes) 000325[0] 1721382072.708154 write_OD 0x160000 thru 0x160fff (0x1000 bytes) 000329[1] 1721382072.724879 mapread 0x057000 thru 0x05c2ff (0x5300 bytes) 000330[0] 1721382072.727423 read_OD 0x081000 thru 0x08ffff (0xf000 bytes) 000331[1] 1721382072.730373 mapwrite 0x0ca000 thru 0x0d4c9e (0xac9f bytes) 000335[0] 1721382072.765785 write_OD 0x4fd000 thru 0x505fff (0x9000 bytes) 000337[1] 1721382072.782137 mapread 0x3d3000 thru 0x3e218d (0xf18e bytes) 000339[1] 1721382072.788766 mapread 0x1c1000 thru 0x1c70ef (0x60f0 bytes) 000340[1] 1721382072.792757 mapwrite 0x1a2000 thru 0x1a9241 (0x7242 bytes) 000341[1] 1721382072.812783 read 0x2eb000 thru 0x2f9a74 (0xea75 bytes) 000342[0] 1721382072.821474 trunc from 0x506000 to 0x932a62 (0x42ca63 bytes) 000343[1] 1721382072.833252 read 0x75e000 thru 0x767753 (0x9754 bytes) 000344[0] 1721382072.846337 mapread 0x186000 thru 0x190a4e (0xaa4f bytes) 000346[1] 1721382072.874050 mapwrite 0x300000 thru 0x3035d6 (0x35d7 bytes) 000347[1] 1721382072.898771 trunc from 0x932a63 to 0x8bc444 (0x7661e bytes) 000349[0] 1721382072.908550 trunc from 0x8bc445 to 0x5865f4 (0x335e50 bytes) 000351[0] 1721382072.917176 write_OD 0x584000 thru 0x585fff (0x2000 bytes) 000352[1] 1721382072.930792 mapread 0x41f000 thru 0x4245ae (0x55af bytes) 000353[0] 1721382072.937313 write_OD 0x76f000 thru 0x777fff (0x9000 bytes) 000354[1] 1721382072.953889 read 0x331000 thru 0x33493e (0x393f bytes) 000355[0] 1721382072.970440 trunc from 0x778000 to 0x5cf7a7 (0x1a8858 bytes) 000356[0] 1721382072.986324 mapwrite 0x59d000 thru 0x5a06a5 (0x36a6 bytes) 000359[1] 1721382073.008015 mapwrite 0x052000 thru 0x05433e (0x233f bytes) 000360[1] 1721382073.036421 trunc from 0x5cf7a8 to 0x209730 (0x3c6077 bytes) 000361[1] 1721382073.043402 mapread 0x04f000 thru 0x05b5b6 (0xc5b7 bytes) 000363[1] 1721382073.061819 mapread 0x15a000 thru 0x1679a9 (0xd9aa bytes) 000366[0] 1721382073.090233 mapwrite 0x06b000 thru 0x072b96 (0x7b97 bytes) 000367[0] 1721382073.116350 mapread 0x056000 thru 0x063a31 (0xda32 bytes) 000369[0] 1721382073.135205 mapwrite 0x158000 thru 0x16199b (0x999c bytes) 000373[1] 1721382073.164463 mapread 0x08d000 thru 0x096662 (0x9663 bytes) 000374[0] 1721382073.177655 read_OD 0x190000 thru 0x199fff (0xa000 bytes) 000376[1] 1721382073.180576 mapread 0x09c000 thru 0x0a8899 (0xc89a bytes) 000377[0] 1721382073.193843 mapwrite 0x872000 thru 0x8724c6 (0x4c7 bytes) 000378[1] 1721382073.242740 write 0x099000 thru 0x09df4b (0x4f4c bytes) 000381[1] 1721382073.259971 read 0x56b000 thru 0x571a6f (0x6a70 bytes) 000382[1] 1721382073.270047 trunc from 0x8724c7 to 0x05639f (0x81c127 bytes) 000385[0] 1721382073.282142 trunc from 0x0563a0 to 0x7ca567 (0x7741c8 bytes) 000388[0] 1721382073.295394 mapwrite 0x059000 thru 0x062dd8 (0x9dd9 bytes) 000390[1] 1721382073.327833 trunc from 0x7ca568 to 0x57d078 (0x24d4ef bytes) 000391[1] 1721382073.336832 read 0x07a000 thru 0x083fc1 (0x9fc2 bytes) 000392[0] 1721382073.355052 read_OD 0x1e2000 thru 0x1e9fff (0x8000 bytes) 000393[0] 1721382073.365201 mapread 0x30a000 thru 0x30a2c0 (0x2c1 bytes) 000395[1] 1721382073.379180 mapwrite 0x1e5000 thru 0x1e8b2a (0x3b2b bytes) 000396[1] 1721382073.402180 trunc from 0x57d079 to 0x23cb85 (0x3404f3 bytes) 000398[1] 1721382073.406753 write 0x99e000 thru 0x9a32d5 (0x52d6 bytes) 000400[0] 1721382073.408479 mapread 0x992000 thru 0x994e7a (0x2e7b bytes) 000402[0] 1721382073.431935 mapwrite 0x56d000 thru 0x56ea80 (0x1a81 bytes) 000404[1] 1721382073.460625 mapread 0x23a000 thru 0x248d94 (0xed95 bytes) 000405[1] 1721382073.490139 read 0x45e000 thru 0x46c673 (0xe674 bytes) 000407[0] 1721382073.497655 read_OD 0x139000 thru 0x143fff (0xb000 bytes) 000408[0] 1721382073.499719 mapread 0x2b4000 thru 0x2b5504 (0x1505 bytes) 000410[0] 1721382073.523567 trunc from 0x9a32d6 to 0x701a51 (0x2a1884 bytes) 000411[0] 1721382073.540913 write_OD 0x7af000 thru 0x7bbfff (0xd000 bytes) 000415[0] 1721382073.559100 mapread 0x4c2000 thru 0x4c2178 (0x179 bytes) 000417[1] 1721382073.560447 mapwrite 0x84a000 thru 0x84dd86 (0x3d87 bytes) 000421[1] 1721382073.593633 mapread 0x00e000 thru 0x016f56 (0x8f57 bytes) 000425[0] 1721382073.608374 write_OD 0x5b3000 thru 0x5bbfff (0x9000 bytes) 000426[1] 1721382073.625428 mapwrite 0x528000 thru 0x537e93 (0xfe94 bytes) 000427[1] 1721382073.661019 read 0x5f6000 thru 0x5ff22d (0x922e bytes) 000430[0] 1721382073.678659 mapwrite 0x6eb000 thru 0x6ebdd2 (0xdd3 bytes) 000435[0] 1721382073.704298 trunc from 0x84dd87 to 0x998325 (0x14a59f bytes) 000436[1] 1721382073.714231 mapread 0x5d9000 thru 0x5da2a0 (0x12a1 bytes) 000437[1] 1721382073.732305 write 0x662000 thru 0x66b9e6 (0x99e7 bytes) 000438[0] 1721382073.738570 mapwrite 0x05b000 thru 0x068fc1 (0xdfc2 bytes) 000441[0] 1721382073.791285 trunc from 0x998326 to 0x5d81e1 (0x3c0144 bytes) 000442[0] 1721382073.797696 mapread 0x3e6000 thru 0x3f0e39 (0xae3a bytes) 000445[1] 1721382073.811216 mapwrite 0x0d8000 thru 0x0e391f (0xb920 bytes) 000446[0] 1721382073.841161 trunc from 0x5d81e2 to 0x3f9032 (0x1df1af bytes) 000448[1] 1721382073.850755 write 0x4a8000 thru 0x4b1e73 (0x9e74 bytes) 000450[1] 1721382073.855835 mapwrite 0x57b000 thru 0x5855e1 (0xa5e2 bytes) 000451[1] 1721382073.891157 write 0x529000 thru 0x53026d (0x726e bytes) 000453[1] 1721382073.892819 mapwrite 0x253000 thru 0x25812d (0x512e bytes) 000454[1] 1721382073.912211 trunc from 0x5855e2 to 0x0ba7b6 (0x4cae2b bytes) 000455[1] 1721382073.915602 read 0x093000 thru 0x0a2c66 (0xfc67 bytes) 000456[1] 1721382073.918298 mapwrite 0x2ee000 thru 0x2f4eb5 (0x6eb6 bytes) 000458[0] 1721382073.944789 mapwrite 0x86c000 thru 0x870bb4 (0x4bb5 bytes) 000459[0] 1721382073.976068 mapread 0x4e9000 thru 0x4f849c (0xf49d bytes) 000461[0] 1721382074.001491 trunc from 0x870bb5 to 0x42af6e (0x445c46 bytes) 000465[0] 1721382074.017025 write_OD 0x189000 thru 0x18efff (0x6000 bytes) 000469[1] 1721382074.032260 trunc from 0x42af6f to 0x547e63 (0x11cef5 bytes) 000471[1] 1721382074.041220 read 0x163000 thru 0x16fed4 (0xced5 bytes) 000473[1] 1721382074.057190 mapwrite 0x892000 thru 0x8959c4 (0x39c5 bytes) 000477[1] 1721382074.100605 mapread 0x35e000 thru 0x364aa7 (0x6aa8 bytes) 000478[0] 1721382074.100982 read_OD 0x765000 thru 0x76efff (0xa000 bytes) 000479[0] 1721382074.111207 mapread 0x1b8000 thru 0x1b9e63 (0x1e64 bytes) 000481[0] 1721382074.119549 trunc from 0x8959c5 to 0x01c34e (0x879676 bytes) 000484[1] 1721382074.133498 read 0x002000 thru 0x0081f5 (0x61f6 bytes) 000491[1] 1721382074.138887 read 0x018000 thru 0x018148 (0x149 bytes) 000493[0] 1721382074.139166 trunc from 0x01c34f to 0x988fe1 (0x96cc93 bytes) 000496[1] 1721382074.148913 trunc from 0x988fe2 to 0x5f1310 (0x397cd1 bytes) 000497[1] 1721382074.153739 read 0x277000 thru 0x280559 (0x955a bytes) 000498[1] 1721382074.172510 read 0x3ec000 thru 0x3f2445 (0x6446 bytes) 000499[0] 1721382074.175559 mapwrite 0x356000 thru 0x3611a3 (0xb1a4 bytes) 000500[1] 1721382074.208928 trunc from 0x5f1311 to 0x5bc3d9 (0x34f37 bytes) All operations completed A-OK! PASS 16a (11s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 05:41:15 (1721382075) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0793029 s, 26.4 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0504505 s, 41.6 MB/s Chance of close/open is 1 in 50 Seed set to 2075 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[1] 1721382075.893902 write 0x19b7bb thru 0x19c526 (0xd6c bytes) 000003[1] 1721382075.900746 read 0x09bd8c thru 0x09bec9 (0x13e bytes) truncating to largest ever: 0x799199 000006[1] 1721382075.908293 trunc from 0x19c527 to 0x799198 (0x5fcc72 bytes) 000009[0] 1721382075.915766 mapread 0x09429c thru 0x094bbf (0x924 bytes) 000010[1] 1721382075.923257 write 0x530947 thru 0x530d47 (0x401 bytes) 000011[1] 1721382075.928665 mapwrite 0x7a443f thru 0x7a63f8 (0x1fba bytes) 000013[1] 1721382075.967262 mapread 0x726fac thru 0x727cb5 (0xd0a bytes) 000015[1] 1721382075.972226 write 0x2fbd1d thru 0x2fc79d (0xa81 bytes) 000022[1] 1721382075.980557 write 0x01eb3d thru 0x01f67b (0xb3f bytes) 000023[1] 1721382075.982451 trunc from 0x7a63f9 to 0x2ece14 (0x4b95e4 bytes) 000025[0] 1721382075.992108 mapread 0x212a9b thru 0x212c6a (0x1d0 bytes) 000027[1] 1721382076.001137 trunc from 0x2ece15 to 0x683eb5 (0x3970a1 bytes) 000028[0] 1721382076.006622 mapread 0x250c43 thru 0x2522b9 (0x1677 bytes) 000033[1] 1721382076.030193 read 0x563d43 thru 0x56559b (0x1859 bytes) 000034[0] 1721382076.040083 mapread 0x2f1946 thru 0x2f2d49 (0x1404 bytes) 000035[0] 1721382076.047819 trunc from 0x683eb6 to 0x37a5dc (0x3098d9 bytes) 000041[0] 1721382076.064366 mapwrite 0x4b6df9 thru 0x4b8309 (0x1511 bytes) 000044[0] 1721382076.094625 read 0x0438ab thru 0x045242 (0x1998 bytes) 000045[0] 1721382076.098489 trunc from 0x4b830a to 0x417845 (0xa0ac4 bytes) 000048[0] 1721382076.104176 trunc from 0x417846 to 0x5d15e3 (0x1b9d9e bytes) 000051[1] 1721382076.114607 mapread 0x0f1cef thru 0x0f32ea (0x15fc bytes) 000052[0] 1721382076.122225 read 0x329daf thru 0x32a997 (0xbe9 bytes) 000053[1] 1721382076.134843 trunc from 0x5d15e4 to 0x374756 (0x25ce8d bytes) 000054[0] 1721382076.145363 trunc from 0x374757 to 0x446ad6 (0xd2380 bytes) 000056[1] 1721382076.158330 write 0x243c61 thru 0x243cba (0x5a bytes) 000058[0] 1721382076.166415 read 0x3f5397 thru 0x3f61fa (0xe64 bytes) 000059[0] 1721382076.172735 write 0x2a6bda thru 0x2a838e (0x17b5 bytes) 000064[0] 1721382076.183891 write 0x874ebc thru 0x876a9c (0x1be1 bytes) 000065[0] 1721382076.185011 mapwrite 0x2a725a thru 0x2a79a5 (0x74c bytes) 000067[0] 1721382076.203109 trunc from 0x876a9d to 0x4f7668 (0x37f434 bytes) 000068[0] 1721382076.210822 trunc from 0x4f7669 to 0x0eec76 (0x4089f2 bytes) 000069[0] 1721382076.219875 mapwrite 0x713ddc thru 0x7156f4 (0x1919 bytes) 000070[1] 1721382076.250718 write 0x10d165 thru 0x10d690 (0x52c bytes) 000072[1] 1721382076.259912 mapwrite 0x8f1d8a thru 0x8f2f5e (0x11d5 bytes) 000074[1] 1721382076.296017 mapwrite 0x5ac8f8 thru 0x5acc28 (0x331 bytes) 000076[1] 1721382076.320896 mapread 0x03480b thru 0x035e42 (0x1638 bytes) 000077[0] 1721382076.323500 mapwrite 0x651b62 thru 0x652c81 (0x1120 bytes) 000078[1] 1721382076.348146 trunc from 0x8f2f5f to 0x3dacb1 (0x5182ad bytes) 000079[0] 1721382076.358123 trunc from 0x3dacb2 to 0x4ecabd (0x111e0c bytes) 000080[1] 1721382076.364062 trunc from 0x4ecabe to 0x48aca6 (0x61e17 bytes) 000081[1] 1721382076.370407 write 0x64d175 thru 0x64dddb (0xc67 bytes) 000082[1] 1721382076.371200 write 0x5229d3 thru 0x523cbd (0x12eb bytes) 000084[1] 1721382076.380574 mapwrite 0x8eded9 thru 0x8ef063 (0x118b bytes) 000086[1] 1721382076.406750 trunc from 0x8ef064 to 0x14762e (0x7a7a35 bytes) 000089[0] 1721382076.415591 mapwrite 0x829a81 thru 0x82a5b7 (0xb37 bytes) 000092[1] 1721382076.450535 trunc from 0x82a5b8 to 0x2f8096 (0x532521 bytes) 000093[1] 1721382076.458031 trunc from 0x2f8097 to 0x07a1cd (0x27dec9 bytes) 000097[0] 1721382076.461629 trunc from 0x07a1ce to 0x014cdc (0x654f1 bytes) 000100[0] 1721382076.468296 write 0x7c31af thru 0x7c3429 (0x27b bytes) 000101[0] 1721382076.470076 write 0x08d999 thru 0x08f8da (0x1f42 bytes) truncating to largest ever: 0x910da4 000103[0] 1721382076.479042 trunc from 0x7c342a to 0x910da3 (0x14d97a bytes) 000104[0] 1721382076.483087 mapwrite 0x0f5d63 thru 0x0f6bce (0xe6c bytes) 000107[1] 1721382076.497211 mapwrite 0x7425b7 thru 0x7444c8 (0x1f12 bytes) 000108[1] 1721382076.525576 read 0x6e782a thru 0x6e9284 (0x1a5b bytes) 000110[0] 1721382076.541785 mapread 0x7f4848 thru 0x7f5bbe (0x1377 bytes) 000111[1] 1721382076.548161 mapwrite 0x25e983 thru 0x26058e (0x1c0c bytes) 000112[1] 1721382076.568446 trunc from 0x910da4 to 0x0d119d (0x83fc06 bytes) 000113[0] 1721382076.573753 write 0x641206 thru 0x641881 (0x67c bytes) 000114[1] 1721382076.579376 mapread 0x610a89 thru 0x611f92 (0x150a bytes) 000117[1] 1721382076.591064 mapwrite 0x750ac7 thru 0x752a68 (0x1fa2 bytes) 000118[0] 1721382076.613917 mapread 0x01f639 thru 0x021625 (0x1fed bytes) 000121[1] 1721382076.625934 read 0x281e9b thru 0x282b28 (0xc8e bytes) 000122[1] 1721382076.645329 trunc from 0x752a69 to 0x26d691 (0x4e53d7 bytes) 000124[0] 1721382076.657731 write 0x2b3c6d thru 0x2b4ff0 (0x1384 bytes) 000125[0] 1721382076.662671 trunc from 0x2b4ff1 to 0x14e2e5 (0x166d0b bytes) 000128[1] 1721382076.672055 read 0x0bdc06 thru 0x0bf7b5 (0x1bb0 bytes) 000129[1] 1721382076.677158 trunc from 0x14e2e6 to 0x4d886d (0x38a588 bytes) 000130[1] 1721382076.683406 mapwrite 0x39172a thru 0x391e28 (0x6ff bytes) 000132[1] 1721382076.708729 write 0x47a76a thru 0x47abc1 (0x458 bytes) 000134[1] 1721382076.713501 mapread 0x141b0b thru 0x14201b (0x511 bytes) 000135[0] 1721382076.713980 trunc from 0x4d886e to 0x00a780 (0x4ce0ed bytes) 000136[1] 1721382076.725752 mapwrite 0x1140dc thru 0x115fe1 (0x1f06 bytes) 000137[0] 1721382076.747068 mapread 0x10757e thru 0x107ffe (0xa81 bytes) 000140[1] 1721382076.755838 mapread 0x0e0015 thru 0x0e07c5 (0x7b1 bytes) 000142[0] 1721382076.759849 mapread 0x0b3ba8 thru 0x0b5320 (0x1779 bytes) 000143[0] 1721382076.766312 read 0x0f91eb thru 0x0fa97d (0x1793 bytes) 000148[1] 1721382076.768993 mapwrite 0x510e0c thru 0x511121 (0x316 bytes) 000149[1] 1721382076.797348 mapread 0x2ac3b3 thru 0x2ad32f (0xf7d bytes) 000150[0] 1721382076.800905 write 0x262757 thru 0x262c70 (0x51a bytes) 000151[1] 1721382076.807062 trunc from 0x511122 to 0x75abd8 (0x249ab7 bytes) 000153[0] 1721382076.815067 write 0x4998ba thru 0x499b97 (0x2de bytes) 000154[1] 1721382076.820985 mapwrite 0x788925 thru 0x789a28 (0x1104 bytes) 000155[1] 1721382076.844496 read 0x3bc832 thru 0x3bd73a (0xf09 bytes) 000156[0] 1721382076.857183 write 0x612a0e thru 0x612a82 (0x75 bytes) 000157[0] 1721382076.864028 mapwrite 0x80f91a thru 0x810a7e (0x1165 bytes) 000158[0] 1721382076.905936 read 0x356276 thru 0x35739a (0x1125 bytes) 000160[0] 1721382076.910236 mapread 0x68a801 thru 0x68bc82 (0x1482 bytes) 000161[0] 1721382076.913646 mapwrite 0x82d799 thru 0x82e08d (0x8f5 bytes) 000163[1] 1721382076.934767 read 0x0dd3c1 thru 0x0de8ac (0x14ec bytes) truncating to largest ever: 0x93ce94 000164[0] 1721382076.943618 trunc from 0x82e08e to 0x93ce93 (0x10ee06 bytes) 000165[1] 1721382076.951805 mapread 0x7bd00e thru 0x7beb4d (0x1b40 bytes) 000167[1] 1721382076.967069 read 0x119eeb thru 0x119fa3 (0xb9 bytes) 000169[1] 1721382076.972663 mapread 0x633800 thru 0x635779 (0x1f7a bytes) 000170[1] 1721382076.979758 mapread 0x480d51 thru 0x481e73 (0x1123 bytes) 000172[1] 1721382076.982569 trunc from 0x93ce94 to 0x37f0ac (0x5bdde7 bytes) 000173[1] 1721382076.994146 mapread 0x09c4a9 thru 0x09c6c3 (0x21b bytes) 000174[1] 1721382076.998352 mapread 0x259169 thru 0x25adbe (0x1c56 bytes) truncating to largest ever: 0x973af0 000176[1] 1721382077.001441 trunc from 0x37f0ad to 0x973aef (0x5f4a43 bytes) 000177[1] 1721382077.008987 mapwrite 0x87a1f2 thru 0x87a4a2 (0x2b1 bytes) 000179[1] 1721382077.024497 read 0x5d020c thru 0x5d11b1 (0xfa6 bytes) 000180[0] 1721382077.038281 read 0x1e8503 thru 0x1ea2a4 (0x1da2 bytes) 000182[1] 1721382077.051038 mapwrite 0x3c3c6b thru 0x3c536e (0x1704 bytes) 000183[1] 1721382077.077532 write 0x89e990 thru 0x89f10d (0x77e bytes) 000185[0] 1721382077.078235 write 0x6cf135 thru 0x6cfa07 (0x8d3 bytes) 000187[1] 1721382077.097890 mapread 0x00e36e thru 0x00eab5 (0x748 bytes) 000191[0] 1721382077.113715 read 0x531df6 thru 0x532032 (0x23d bytes) 000198[1] 1721382077.127854 read 0x06b9f8 thru 0x06c38e (0x997 bytes) 000199[1] 1721382077.151342 write 0x0078bc thru 0x007f6e (0x6b3 bytes) 000201[1] 1721382077.162193 mapwrite 0x1928bb thru 0x19345c (0xba2 bytes) 000203[0] 1721382077.194022 read 0x88dd1f thru 0x88f00b (0x12ed bytes) 000204[0] 1721382077.202177 read 0x310fec thru 0x311d63 (0xd78 bytes) 000205[1] 1721382077.219548 read 0x3808ab thru 0x380922 (0x78 bytes) 000210[0] 1721382077.234389 mapwrite 0x7b81b8 thru 0x7ba0d0 (0x1f19 bytes) 000213[1] 1721382077.274201 write 0x604ced thru 0x6056ae (0x9c2 bytes) 000215[0] 1721382077.285407 read 0x666b4b thru 0x666cb1 (0x167 bytes) 000217[0] 1721382077.304242 mapwrite 0x44d2f9 thru 0x44e0a6 (0xdae bytes) 000218[1] 1721382077.320296 read 0x7d7859 thru 0x7d787f (0x27 bytes) 000219[1] 1721382077.325668 trunc from 0x973af0 to 0x39b0db (0x5d8a14 bytes) 000223[1] 1721382077.340732 write 0x788dc4 thru 0x78a7c5 (0x1a02 bytes) 000225[1] 1721382077.345285 write 0x92c663 thru 0x92c9b8 (0x356 bytes) 000229[0] 1721382077.355092 mapread 0x1129b6 thru 0x113321 (0x96c bytes) 000232[1] 1721382077.366118 mapread 0x3dc3f2 thru 0x3dcca4 (0x8b3 bytes) 000234[1] 1721382077.374223 mapwrite 0x944139 thru 0x94521f (0x10e7 bytes) 000236[0] 1721382077.400151 write 0x77894a thru 0x77a7e4 (0x1e9b bytes) 000237[0] 1721382077.406565 trunc from 0x945220 to 0x1813f2 (0x7c3e2d bytes) 000238[1] 1721382077.412077 read 0x0400c1 thru 0x041ba6 (0x1ae6 bytes) 000240[0] 1721382077.425982 read 0x0688a5 thru 0x069a3a (0x1196 bytes) 000241[0] 1721382077.434984 trunc from 0x1813f3 to 0x700cb6 (0x57f8c4 bytes) 000242[1] 1721382077.445084 write 0x6ce3c9 thru 0x6cf22b (0xe63 bytes) 000243[0] 1721382077.451546 read 0x457794 thru 0x45828a (0xaf7 bytes) 000244[1] 1721382077.465523 trunc from 0x700cb7 to 0x54fc56 (0x1b1060 bytes) 000249[1] 1721382077.479080 mapwrite 0x2a3d84 thru 0x2a5b4e (0x1dcb bytes) 000250[0] 1721382077.508338 trunc from 0x54fc57 to 0x91f197 (0x3cf541 bytes) 000252[0] 1721382077.525212 trunc from 0x91f198 to 0x47c33e (0x4a2e59 bytes) 000256[1] 1721382077.536228 read 0x24a3f7 thru 0x24ac9c (0x8a6 bytes) 000259[0] 1721382077.550196 write 0x1f294e thru 0x1f29d3 (0x86 bytes) 000263[1] 1721382077.551921 trunc from 0x47c33f to 0x716697 (0x29a359 bytes) 000264[1] 1721382077.564518 read 0x0709f3 thru 0x070ac1 (0xcf bytes) 000265[1] 1721382077.586017 mapwrite 0x438352 thru 0x439c45 (0x18f4 bytes) 000266[1] 1721382077.601511 write 0x6ccb2f thru 0x6ce400 (0x18d2 bytes) 000267[1] 1721382077.604901 mapwrite 0x6f6552 thru 0x6f7e96 (0x1945 bytes) 000268[0] 1721382077.621524 mapread 0x07d0c8 thru 0x07dd97 (0xcd0 bytes) 000269[1] 1721382077.630741 mapread 0x6275b0 thru 0x62840e (0xe5f bytes) 000271[0] 1721382077.635728 trunc from 0x716698 to 0x8f9bd2 (0x1e353b bytes) 000273[0] 1721382077.656212 trunc from 0x8f9bd3 to 0x58b021 (0x36ebb1 bytes) 000275[0] 1721382077.667772 mapwrite 0x2a5e05 thru 0x2a5ed3 (0xcf bytes) 000278[1] 1721382077.688633 trunc from 0x58b022 to 0x17d129 (0x40def8 bytes) 000279[0] 1721382077.695620 trunc from 0x17d12a to 0x6e880d (0x56b6e4 bytes) 000280[0] 1721382077.704257 write 0x3e0c32 thru 0x3e16f8 (0xac7 bytes) 000282[1] 1721382077.710587 read 0x585ac7 thru 0x586e81 (0x13bb bytes) 000283[0] 1721382077.720780 write 0x4a005d thru 0x4a1513 (0x14b7 bytes) 000284[0] 1721382077.723598 write 0x524430 thru 0x525d33 (0x1904 bytes) 000286[0] 1721382077.730825 trunc from 0x6e880e to 0x694afb (0x53d12 bytes) 000287[1] 1721382077.734712 mapwrite 0x16bce4 thru 0x16c35c (0x679 bytes) 000289[1] 1721382077.763042 trunc from 0x694afc to 0x16071c (0x5343df bytes) 000293[1] 1721382077.777725 mapwrite 0x7955e2 thru 0x79660c (0x102b bytes) 000295[1] 1721382077.811573 mapread 0x733642 thru 0x7354eb (0x1eaa bytes) 000296[0] 1721382077.812075 write 0x5f13f8 thru 0x5f18df (0x4e8 bytes) 000298[1] 1721382077.825352 mapwrite 0x5035b4 thru 0x5041b7 (0xc04 bytes) 000301[1] 1721382077.849172 write 0x8a23ea thru 0x8a3919 (0x1530 bytes) 000302[1] 1721382077.849863 mapread 0x1a216e thru 0x1a311c (0xfaf bytes) 000304[0] 1721382077.857560 trunc from 0x8a391a to 0x301e4e (0x5a1acb bytes) 000305[1] 1721382077.868708 trunc from 0x301e4f to 0x35659e (0x54750 bytes) 000306[0] 1721382077.879082 read 0x2fe08a thru 0x2ffee2 (0x1e59 bytes) 000308[1] 1721382077.886298 mapread 0x03d468 thru 0x03e112 (0xcab bytes) 000309[0] 1721382077.891279 mapread 0x2bb4bf thru 0x2bd155 (0x1c97 bytes) 000310[0] 1721382077.898642 read 0x18a1c8 thru 0x18a673 (0x4ac bytes) 000311[1] 1721382077.908030 mapread 0x13a034 thru 0x13b93e (0x190b bytes) 000314[1] 1721382077.910418 trunc from 0x35659f to 0x90ab92 (0x5b45f4 bytes) 000315[1] 1721382077.925124 trunc from 0x90ab93 to 0x202a64 (0x70812e bytes) 000318[0] 1721382077.935705 read 0x0c75ef thru 0x0c8c99 (0x16ab bytes) 000319[0] 1721382077.944152 mapread 0x04735d thru 0x048b81 (0x1825 bytes) 000323[0] 1721382077.955140 mapread 0x16201d thru 0x1632a5 (0x1289 bytes) 000324[1] 1721382077.964799 write 0x2c18e4 thru 0x2c2151 (0x86e bytes) 000329[1] 1721382077.976122 write 0x697bf5 thru 0x6996f2 (0x1afe bytes) 000330[0] 1721382077.977249 mapwrite 0x072e00 thru 0x07302f (0x230 bytes) 000331[0] 1721382077.997527 write 0x35d57f thru 0x35e00a (0xa8c bytes) 000334[0] 1721382078.008443 mapwrite 0x3d9e54 thru 0x3daa4d (0xbfa bytes) 000335[1] 1721382078.024842 mapwrite 0x639b5a thru 0x63b86b (0x1d12 bytes) 000337[1] 1721382078.052158 mapwrite 0x259257 thru 0x25a0f6 (0xea0 bytes) 000339[1] 1721382078.072322 write 0x62056c thru 0x620eed (0x982 bytes) 000341[1] 1721382078.078549 trunc from 0x6996f3 to 0x5da167 (0xbf58b bytes) 000342[1] 1721382078.087450 read 0x582fc0 thru 0x583587 (0x5c8 bytes) 000344[1] 1721382078.094939 read 0x59703f thru 0x59710d (0xcf bytes) 000345[1] 1721382078.095195 mapread 0x35005f thru 0x35134b (0x12ed bytes) 000346[1] 1721382078.098389 write 0x711516 thru 0x712cba (0x17a5 bytes) 000348[0] 1721382078.104699 write 0x67a84e thru 0x67b4f7 (0xcaa bytes) 000349[1] 1721382078.109527 mapwrite 0x65839a thru 0x659f72 (0x1bd9 bytes) 000352[0] 1721382078.132721 mapread 0x010d92 thru 0x0122c0 (0x152f bytes) 000353[1] 1721382078.141596 trunc from 0x712cbb to 0x464bd8 (0x2ae0e2 bytes) 000355[0] 1721382078.147221 mapread 0x1fae68 thru 0x1faed1 (0x6a bytes) 000356[1] 1721382078.154730 read 0x45b2f4 thru 0x45bf42 (0xc4f bytes) 000358[0] 1721382078.160021 mapwrite 0x3b79d7 thru 0x3b8add (0x1107 bytes) 000359[1] 1721382078.181394 mapwrite 0x820f61 thru 0x8217f6 (0x896 bytes) 000361[1] 1721382078.215316 trunc from 0x8217f7 to 0x62a559 (0x1f729d bytes) 000362[0] 1721382078.219886 mapwrite 0x299368 thru 0x299e02 (0xa9b bytes) 000364[0] 1721382078.242524 trunc from 0x62a55a to 0x098c5b (0x5918fe bytes) 000365[0] 1721382078.251670 mapwrite 0x4518e5 thru 0x452d20 (0x143c bytes) 000366[0] 1721382078.272308 mapwrite 0x3e8a63 thru 0x3e9f8c (0x152a bytes) 000367[0] 1721382078.290317 mapwrite 0x8da310 thru 0x8daefe (0xbef bytes) 000369[0] 1721382078.314753 mapread 0x3fe1a9 thru 0x3feae4 (0x93c bytes) 000372[0] 1721382078.319055 mapread 0x1af469 thru 0x1afa9c (0x634 bytes) 000374[1] 1721382078.323633 mapwrite 0x465aa9 thru 0x46788d (0x1de5 bytes) 000375[0] 1721382078.343843 trunc from 0x8daeff to 0x193b58 (0x7473a6 bytes) 000378[1] 1721382078.357759 mapwrite 0x5f6219 thru 0x5f6c0f (0x9f7 bytes) 000380[0] 1721382078.382813 write 0x4bcbf8 thru 0x4be8eb (0x1cf4 bytes) 000383[0] 1721382078.394931 mapread 0x1482e2 thru 0x1484d1 (0x1f0 bytes) 000384[0] 1721382078.399536 mapwrite 0x52f118 thru 0x530674 (0x155d bytes) 000385[1] 1721382078.418411 read 0x11b709 thru 0x11d59e (0x1e96 bytes) 000387[1] 1721382078.430670 trunc from 0x5f6c10 to 0x708587 (0x111978 bytes) 000388[0] 1721382078.440874 trunc from 0x708588 to 0x05adb6 (0x6ad7d1 bytes) 000389[0] 1721382078.450811 write 0x29ab55 thru 0x29ba82 (0xf2e bytes) 000392[1] 1721382078.462516 mapwrite 0x1eb791 thru 0x1ec6b6 (0xf26 bytes) 000393[1] 1721382078.498797 read 0x06fb6e thru 0x06fc25 (0xb8 bytes) 000396[1] 1721382078.514777 write 0x6585a3 thru 0x658ec3 (0x921 bytes) 000398[0] 1721382078.518641 trunc from 0x658ec4 to 0x321a63 (0x337460 bytes) 000399[1] 1721382078.526803 mapwrite 0x7d0c50 thru 0x7d0ef1 (0x2a2 bytes) 000401[1] 1721382078.570497 write 0x3b26a4 thru 0x3b2dd1 (0x72e bytes) truncating to largest ever: 0x9764c6 000403[1] 1721382078.576069 trunc from 0x7d0ef2 to 0x9764c5 (0x1a55d4 bytes) 000404[1] 1721382078.579681 mapwrite 0x43fd6d thru 0x4415be (0x1852 bytes) 000407[1] 1721382078.601399 read 0x0025e6 thru 0x0034be (0xed9 bytes) 000408[1] 1721382078.615829 read 0x4cc334 thru 0x4cd104 (0xdd1 bytes) 000409[1] 1721382078.618829 write 0x90c898 thru 0x90dc23 (0x138c bytes) 000410[1] 1721382078.621734 trunc from 0x9764c6 to 0x231773 (0x744d52 bytes) 000412[1] 1721382078.637542 trunc from 0x231774 to 0x16b141 (0xc6632 bytes) 000413[0] 1721382078.641757 mapwrite 0x037517 thru 0x038037 (0xb21 bytes) 000414[0] 1721382078.664677 trunc from 0x16b142 to 0x8ec8c1 (0x781780 bytes) 000415[0] 1721382078.672095 mapwrite 0x589804 thru 0x58b461 (0x1c5e bytes) 000417[0] 1721382078.692909 mapread 0x1d644e thru 0x1d7d33 (0x18e6 bytes) 000418[0] 1721382078.694903 read 0x15ae2c thru 0x15b1e5 (0x3ba bytes) 000419[1] 1721382078.712379 mapread 0x7278d7 thru 0x7297ea (0x1f14 bytes) 000420[1] 1721382078.723651 trunc from 0x8ec8c2 to 0x869869 (0x83058 bytes) 000421[1] 1721382078.736240 mapwrite 0x675326 thru 0x675724 (0x3ff bytes) 000422[1] 1721382078.753417 write 0x545fd2 thru 0x546cc6 (0xcf5 bytes) 000423[0] 1721382078.756343 trunc from 0x86986a to 0x38b77e (0x4de0eb bytes) 000424[1] 1721382078.764115 trunc from 0x38b77f to 0x05fa3d (0x32bd41 bytes) 000426[0] 1721382078.777529 mapwrite 0x974bb2 thru 0x975f27 (0x1376 bytes) 000427[0] 1721382078.809927 trunc from 0x975f28 to 0x076276 (0x8ffcb1 bytes) 000428[0] 1721382078.814316 trunc from 0x076277 to 0x074620 (0x1c56 bytes) 000430[0] 1721382078.821396 write 0x28d52c thru 0x28f0bc (0x1b91 bytes) 000431[0] 1721382078.822204 write 0x4fb3f7 thru 0x4fb789 (0x393 bytes) 000434[0] 1721382078.823201 mapwrite 0x62daf2 thru 0x62f91c (0x1e2b bytes) 000435[1] 1721382078.846157 mapwrite 0x2e70f8 thru 0x2e7179 (0x82 bytes) 000437[0] 1721382078.871146 mapread 0x589f2d thru 0x58b2d8 (0x13ac bytes) 000438[1] 1721382078.878160 write 0x04b9bc thru 0x04c044 (0x689 bytes) 000441[0] 1721382078.889743 trunc from 0x62f91d to 0x2e24fd (0x34d41f bytes) 000442[1] 1721382078.901811 trunc from 0x2e24fe to 0x601b4e (0x31f651 bytes) 000446[0] 1721382078.917131 mapwrite 0x19f67c thru 0x19ff09 (0x88e bytes) 000447[0] 1721382078.937621 mapwrite 0x5f9dd2 thru 0x5fad1b (0xf4a bytes) 000449[0] 1721382078.961532 read 0x06616a thru 0x068167 (0x1ffe bytes) 000454[0] 1721382078.985346 mapwrite 0x29531a thru 0x296b2f (0x1816 bytes) 000455[1] 1721382079.014379 write 0x7c0455 thru 0x7c085b (0x407 bytes) 000456[1] 1721382079.022913 read 0x6d2cf5 thru 0x6d3a1f (0xd2b bytes) 000457[1] 1721382079.030978 read 0x2ac8f6 thru 0x2ae595 (0x1ca0 bytes) 000458[1] 1721382079.047068 mapwrite 0x602ccb thru 0x602e6b (0x1a1 bytes) 000459[0] 1721382079.077561 write 0x1e5bc5 thru 0x1e70e8 (0x1524 bytes) 000460[0] 1721382079.089281 mapwrite 0x8530c4 thru 0x853377 (0x2b4 bytes) 000462[0] 1721382079.113978 trunc from 0x853378 to 0x5c22ee (0x291089 bytes) 000464[0] 1721382079.121816 write 0x0f5c34 thru 0x0f7445 (0x1812 bytes) 000465[1] 1721382079.123981 read 0x4b3766 thru 0x4b55ec (0x1e87 bytes) 000471[1] 1721382079.141789 mapwrite 0x736506 thru 0x736c7d (0x778 bytes) 000472[1] 1721382079.186817 mapwrite 0x394f69 thru 0x3957db (0x873 bytes) 000474[0] 1721382079.211638 write 0x8272e7 thru 0x829079 (0x1d93 bytes) 000477[1] 1721382079.224672 mapwrite 0x4d4f86 thru 0x4d637f (0x13fa bytes) 000478[0] 1721382079.249860 read 0x73cde1 thru 0x73d83a (0xa5a bytes) 000479[1] 1721382079.259107 mapread 0x224dd6 thru 0x2266f7 (0x1922 bytes) 000481[0] 1721382079.263232 mapwrite 0x9d826e thru 0x9d8e51 (0xbe4 bytes) 000482[0] 1721382079.294433 mapwrite 0x155c6a thru 0x156960 (0xcf7 bytes) 000486[1] 1721382079.315092 mapread 0x0a0a59 thru 0x0a1c84 (0x122c bytes) 000487[1] 1721382079.332436 read 0x3d2735 thru 0x3d27b3 (0x7f bytes) 000488[1] 1721382079.349680 read 0x177606 thru 0x1794c5 (0x1ec0 bytes) 000489[1] 1721382079.357917 read 0x842a81 thru 0x8448e7 (0x1e67 bytes) 000491[0] 1721382079.358410 mapread 0x46d712 thru 0x46e1a6 (0xa95 bytes) 000493[0] 1721382079.361772 read 0x1768dd thru 0x1784a7 (0x1bcb bytes) 000495[1] 1721382079.379307 trunc from 0x9d8e52 to 0x0f6bbd (0x8e2294 bytes) 000496[1] 1721382079.398320 read 0x0cb151 thru 0x0cc2b8 (0x1168 bytes) 000497[0] 1721382079.401841 write 0x928726 thru 0x929982 (0x125d bytes) 000498[0] 1721382079.409708 trunc from 0x929983 to 0x4a6ddb (0x482ba7 bytes) All operations completed A-OK! PASS 16b (5s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 05:41:20 (1721382080) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 05:41:21 (1721382081) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0888037 s, 23.6 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0577311 s, 36.3 MB/s ldlm.namespaces.MGC192.168.203.128@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b5884000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a5e9800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b5884000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b5884000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.7427 s, 38.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.622192 s, 169 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.588925 s, 178 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.12707 s, 93.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.76559 s, 59.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.878104 s, 119 MB/s PASS 16d (10s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 05:41:31 (1721382091) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0598268 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0312372 s, 33.6 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 05:41:33 (1721382093) TIMEOUT PASS 16f (21s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 05:41:54 (1721382114) TIMEOUT PASS 16g (22s) == sanityn test 16i: read after truncate file ============ 05:42:16 (1721382136) ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (2s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 05:42:18 (1721382138) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 05:42:22 (1721382142) mmap test1: basic mmap operation (PASS, 0.020124s) mmap test2: MAP_PRIVATE not write back (PASS, 0.014605s) mmap test3: concurrent mmap ops on two nodes (PASS, 3.4467s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 2.0834s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.022366s) mmap test6: check mmap write/read content on two nodes (PASS, 0.027798s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0121s) PASS 18 (9s) == sanityn test 19: test concurrent uncached read races ========================================================================= 05:42:31 (1721382151) oleg328-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 05:42:32 (1721382152) PASS 20 (1s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 05:42:33 (1721382153) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 05:42:35 (1721382155) now is 1721382155 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 new atime is 1721382216 PASS 23 (62s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 05:43:38 (1721382218) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.5K 368 488.1K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 376 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 230.1K 368 229.8K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.5K 368 488.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 376 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 368 229.8K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 13.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500203 368 499835 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118008 376 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235664 368 235296 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.5K 368 488.1K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 376 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 368 229.8K 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500203 368 499835 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118008 376 117632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre[OST:1] filesystem_summary: 235664 368 235296 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500203 368 499835 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118008 376 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235664 368 235296 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 13312 7525376 1% /mnt/lustre2 PASS 24a (1s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 05:43:39 (1721382219) PASS 24b (1s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 05:43:40 (1721382220) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 05:43:42 (1721382222) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 05:43:43 (1721382223) utime: good mknod times 1721382222 <= 1721382223 <= 1721382223 for /mnt/lustre/f26a utime: good mknod times 1721382222 <= 1721382223 <= 1721382223 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (2s) == sanityn test 26b: sync mtime between ost and mds ====== 05:43:45 (1721382225) PASS 26b (4s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 05:43:49 (1721382229) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 05:43:53 (1721382233) fail_loc=0x314 PASS 31a (2s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 05:43:55 (1721382235) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg328-server mds-ost sync done. sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.128@tcp:/lustre 7542784 14336 7524352 1% /mnt/lustre2 PASS 31b (16s) == sanityn test 31r: open-rename(replace) race =========== 05:44:12 (1721382252) -rw-r--r-- 1 root root 0 Jul 19 05:44 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 05:44:16 (1721382256) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel ========================================================== 05:44:17 (1721382257) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: DNE distributed operation should trigger COS ========================================================== 05:44:18 (1721382258) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (0s) == sanityn test 33e: DNE local operation shouldn't trigger COS ========================================================== 05:44:18 (1721382258) SKIP: sanityn test_33e Need two or more clients, have 1 SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 05:44:19 (1721382259) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid in IDLE state after 0 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid in FULL state after 0 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid in IDLE state after 0 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid in FULL state after 0 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800b5884000.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff88012a5e9800.ost_server_uuid in IDLE state after 0 sec oleg328-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid,osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800b5884000.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff88012a5e9800.ost_server_uuid in IDLE state after 0 sec PASS 34 (71s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 05:45:30 (1721382330) fail_loc=0x80000317 Race attempt 0 Wait for 15492 15636 for 60 sec... total: 4000 open/close in 7.34 seconds: 545.11 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.128@tcp:/lustre 7.2G 14M 7.2G 1% /mnt/lustre 192.168.203.128@tcp:/lustre 7.2G 14M 7.2G 1% /mnt/lustre2 PASS 35 (64s) == sanityn test 36: handle ESTALE/open-unlink correctly == 05:46:34 (1721382394) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg328-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.24678 s, 212 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(0) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.260467 s, 201 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(1) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.271771 s, 193 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(2) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.309855 s, 169 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(3) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.290223 s, 181 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(4) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.344333 s, 152 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(5) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.268813 s, 195 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(6) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.309066 s, 170 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(7) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.335834 s, 156 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(8) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.293477 s, 179 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(9) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.268023 s, 196 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(10) *** before(7524352) after_dd(7473152) after(7524352) PASS 36 (227s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 05:50:21 (1721382621) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 total: 10000 create in 9.34 seconds: 1070.29 ops/second PASS 37 (12s) == sanityn test 39a: test from 11063 ============================================================================================ 05:50:33 (1721382633) repeat after cancel_lru_locks PASS 39a (2s) == sanityn test 39b: 11063 problem 1 ============================================================================================ 05:50:35 (1721382635) repeat after cancel_lru_locks PASS 39b (2s) == sanityn test 39c: check truncate mtime update ================================================================================ 05:50:37 (1721382637) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 05:50:40 (1721382640) fail_loc=0x411 fail_loc=0 PASS 39d (1s) == sanityn test 40a: pdirops: create vs others ======================================================================== 05:50:41 (1721382641) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (7s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 05:50:48 (1721382648) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (7s) == sanityn test 40c: pdirops: link and others ======================================================================== 05:50:55 (1721382655) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 05:51:02 (1721382662) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 05:51:09 (1721382669) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (6s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 05:51:15 (1721382675) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (4s) == sanityn test 41b: pdirops: create vs create ======================================================================== 05:51:19 (1721382679) open: File exists Conflict PASS 41b (4s) == sanityn test 41c: pdirops: create vs link ======================================================================== 05:51:23 (1721382683) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 05:51:28 (1721382688) Conflict PASS 41d (4s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 05:51:32 (1721382692) Conflict PASS 41e (4s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 05:51:36 (1721382696) Conflict PASS 41f (4s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 05:51:40 (1721382700) Conflict PASS 41g (5s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 05:51:45 (1721382705) Conflict PASS 41h (4s) == sanityn test 41i: reint_open: create vs create ======== 05:51:49 (1721382709) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (661s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 06:02:50 (1721383370) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 06:02:55 (1721383375) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 06:03:00 (1721383380) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 06:03:06 (1721383386) Conflict PASS 42d (4s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 06:03:10 (1721383390) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 06:03:15 (1721383395) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 06:03:20 (1721383400) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 06:03:25 (1721383405) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 06:03:30 (1721383410) PASS 43a (17s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 06:03:47 (1721383427) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 06:03:53 (1721383433) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 06:03:58 (1721383438) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 06:04:03 (1721383443) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 06:04:08 (1721383448) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (6s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 06:04:14 (1721383454) stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 06:04:19 (1721383459) Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 06:04:24 (1721383464) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (0s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 06:04:25 (1721383465) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (47s) == sanityn test 43k: unlink vs create ==================== 06:05:12 (1721383512) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (380s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 06:11:32 (1721383892) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Jul 19 06:11:35 EDT 2024 PASS 44a (4s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 06:11:36 (1721383896) open: File exists Conflict PASS 44b (4s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 06:11:40 (1721383900) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (5s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 06:11:45 (1721383905) Conflict PASS 44d (4s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 06:11:49 (1721383909) Conflict PASS 44e (4s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 06:11:53 (1721383913) Conflict PASS 44f (4s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 06:11:57 (1721383917) Conflict PASS 44g (4s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 06:12:02 (1721383922) Conflict PASS 44h (4s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 06:12:06 (1721383926) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 06:12:07 (1721383927) PASS 45a (18s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 06:12:25 (1721383945) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 06:12:30 (1721383950) Conflict PASS 45c (4s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 06:12:34 (1721383954) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (4s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 06:12:38 (1721383958) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 06:12:43 (1721383963) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (4s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 06:12:47 (1721383967) stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (4s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 06:12:51 (1721383971) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 06:12:55 (1721383975) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (0s) == sanityn test 45j: read vs rename ====================== 06:12:56 (1721383976) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (367s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 06:19:03 (1721384343) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (4s) == sanityn test 46b: pdirops: link vs create ======================================================================== 06:19:07 (1721384347) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 06:19:12 (1721384352) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (4s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 06:19:16 (1721384356) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 06:19:21 (1721384361) Conflict PASS 46e (4s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 06:19:25 (1721384365) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 06:19:30 (1721384370) Conflict PASS 46g (4s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 06:19:34 (1721384374) Conflict PASS 46h (4s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 06:19:38 (1721384378) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 06:19:39 (1721384379) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 06:19:40 (1721384380) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 06:19:41 (1721384381) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 06:19:42 (1721384382) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 06:19:43 (1721384383) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (0s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 06:19:43 (1721384383) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 06:19:44 (1721384384) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (1s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 06:19:45 (1721384385) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00486933 s, 2.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (6s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 06:19:51 (1721384391) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.00274086 s, 58.0 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3175: kill: (24511) - No such process PASS 51a (4s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 06:19:55 (1721384395) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00190754 s, 537 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114388 s, 91.7 MB/s 3145728 PASS 51b (13s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 06:20:08 (1721384408) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00320249 s, 320 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 06:20:14 (1721384414) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895441 s, 117 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 06:20:17 (1721384417) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2437 0x985 0 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2437 0x985 0 read lease(1) has applied. PASS 51e (3s) == sanityn test 54: rename locking ======================= 06:20:20 (1721384420) llite.lustre-ffff8800b5884000.xattr_cache=1 llite.lustre-ffff88012a5e9800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504963 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:21.000000000 -0400 Modify: 2024-07-19 06:20:21.000000000 -0400 Change: 2024-07-19 06:20:21.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504966 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:26.000000000 -0400 Modify: 2024-07-19 06:20:27.000000000 -0400 Change: 2024-07-19 06:20:27.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504969 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:32.000000000 -0400 Modify: 2024-07-19 06:20:32.000000000 -0400 Change: 2024-07-19 06:20:32.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504968 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:32.000000000 -0400 Modify: 2024-07-19 06:20:32.000000000 -0400 Change: 2024-07-19 06:20:32.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504971 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:37.000000000 -0400 Modify: 2024-07-19 06:20:37.000000000 -0400 Change: 2024-07-19 06:20:37.000000000 -0400 Birth: - File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504972 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:20:37.000000000 -0400 Modify: 2024-07-19 06:20:37.000000000 -0400 Change: 2024-07-19 06:20:37.000000000 -0400 Birth: - PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 06:20:44 (1721384444) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (6s) == sanityn test 55b: rename vs unlink source dir ========= 06:20:50 (1721384450) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 06:20:57 (1721384457) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 06:21:09 (1721384469) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (9s) == sanityn test 60: Verify data_version behaviour ======== 06:21:18 (1721384478) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 06:21:20 (1721384480) PASS 70a (1s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 06:21:21 (1721384481) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 06:21:23 (1721384483) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00274251 s, 23.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 06:21:24 (1721384484) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00585134 s, 11.2 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 06:21:25 (1721384485) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (1s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 06:21:26 (1721384486) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00463746 s, 14.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 06:21:27 (1721384487) llite.lustre-ffff8800b5884000.xattr_cache=1 llite.lustre-ffff88012a5e9800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (1s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 06:21:28 (1721384488) llite.lustre-ffff8800b5884000.xattr_cache=1 llite.lustre-ffff88012a5e9800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (2s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 06:21:30 (1721384490) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.0027083 s, 378 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00223785 s, 458 kB/s 3166: get lock2 3167: get lock1 3166: done 3166: get lock1 3167: done 3167: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 3166: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 3167: cannot get lock2: Resource deadlock avoided 3167: exit rc=0 3166: done 3166: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 06:21:36 (1721384496) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0161862 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900055 s, 117 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3743: 3801 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (7s) == sanityn test 76: Verify MDT open_files listing ======== 06:21:43 (1721384503) opening files: ........ closing files: ........ PASS 76 (15s) == sanityn test 77a: check FIFO NRS policy =============== 06:21:58 (1721384518) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.120464 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00689646 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189679 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194946 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168935 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244844 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262112 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.021741 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272506 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0300027 s, 34.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0185088 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273835 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251489 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224686 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203659 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178002 s, 58.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014386 s, 72.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00897287 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00198829 s, 527 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00786455 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00770446 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100192 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00154009 s, 681 MB/s 1048576 bytes (1.0 MB) copied, 0.00476776 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00170535 s, 615 MB/s , 0.00967592 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00957833 s, 109 MB/s 1048576 bytes (1.0 MB) copied, 0.0100143 s, 105 MB/s , 0.00153452 s, 683 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00989621 s, 106 MB/s , 0.0103992 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128758 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104075 s, 101 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 06:22:00 (1721384520) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.10649 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00862199 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153347 s, 68.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0094638 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00969293 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178439 s, 58.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0203684 s, 51.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0125431 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.027324 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112993 s, 92.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0225169 s, 46.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0427474 s, 24.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325337 s, 32.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0451763 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0539149 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0481173 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0544399 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00201554 s, 520 MB/s , 0.00699036 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00669189 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00593704 s, 177 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00743206 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106146 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00689356 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00832971 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893936 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00130737 s, 802 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00122836 s, 854 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135335 s, 77.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104738 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0011448 s, 916 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110162 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107052 s, 98.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=2 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.101086 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00787494 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169264 s, 61.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226872 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256248 s, 40.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394434 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0326929 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332396 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316821 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0203735 s, 51.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0297937 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036169 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277735 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.036443 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419844 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280298 s, 37.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039556 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956264 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884887 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00714389 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00157118 s, 667 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00788742 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00787691 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0114645 s, 91.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0136092 s, 77.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993157 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012111 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099422 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00155085 s, 676 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0015447 s, 679 MB/s 1048576 bytes (1.0 MB) copied, 0.0114661 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124056 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126212 s, 83.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 06:22:03 (1721384523) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.111699 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00911045 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00949928 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215949 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260003 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0222393 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186067 s, 56.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249592 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0303912 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262407 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161658 s, 64.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239929 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0372165 s, 28.2 MB/s , 0.0162282 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0395326 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0352593 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237795 s, 44.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00940132 s, 112 MB/s , 0.0092013 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00134555 s, 779 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00747304 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00544026 s, 193 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752434 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00172006 s, 610 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0017201 s, 610 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00148307 s, 707 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161209 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00692333 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00145327 s, 722 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00163963 s, 640 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00349942 s, 300 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00208981 s, 502 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00168424 s, 623 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0990978 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00764256 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00805981 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154329 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0176661 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0243369 s, 43.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0127319 s, 82.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141881 s, 73.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0179026 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174121 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145123 s, 72.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.035235 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317887 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0482363 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0392534 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0520896 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0602084 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00802955 s, 131 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00681062 s, 154 MB/s , 0.00614151 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00687088 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00750296 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00692294 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128761 s, 81.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00748944 s, 140 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00788192 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00941176 s, 111 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00385726 s, 272 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00196523 s, 534 MB/s , 0.00341725 s, 307 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00137609 s, 762 MB/s 1048576 bytes (1.0 MB) copied, 0.00734675 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00188002 s, 558 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=3 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.100831 s, 166 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812062 s, 129 MB/s 1048576 bytes (1.0 MB) copied, 0.0114668 s, 91.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156211 s, 67.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100946 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010301 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167498 s, 62.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00861694 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131361 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0080329 s, 131 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0401131 s, 26.1 MB/s 1048576 bytes (1.0 MB) copied, 0.040362 s, 26.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0422935 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0344644 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0364805 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339385 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0488655 s, 21.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00839819 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0083345 s, 126 MB/s , 0.00657305 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824152 s, 127 MB/s 1048576 bytes (1.0 MB) copied, 0.0088214 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00822472 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125144 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884006 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956106 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0101661 s, 103 MB/s 1048576 bytes (1.0 MB) copied, 0.0105387 s, 99.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116302 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122972 s, 85.3 MB/s 1048576 bytes (1.0 MB) copied, 0.012763 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105688 s, 99.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117899 s, 88.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (4s) == sanityn test 77d: check TRR nrs policy ================ 06:22:07 (1721384527) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.156355 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00784867 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00887204 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252726 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253983 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272216 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240398 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259539 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254156 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.024477 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0276765 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239447 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329279 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250546 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332827 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318331 s, 32.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200853 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00831257 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00727726 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00734534 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00998597 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00906771 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104198 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00553582 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00516998 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899113 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00912945 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010956 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104964 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105143 s, 99.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010572 s, 99.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.010592 s, 99.0 MB/s , 0.0119722 s, 87.6 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.104917 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843783 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200826 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0282998 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249451 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317985 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235289 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287019 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243991 s, 43.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319154 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165566 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0249082 s, 42.1 MB/s , 0.0298779 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226156 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0272413 s, 38.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0305152 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140502 s, 74.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00750478 s, 140 MB/s , 0.00589524 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00525745 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752348 s, 139 MB/s , 0.00652841 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00673698 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00598654 s, 175 MB/s , 0.00118205 s, 887 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0058121 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099254 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00119411 s, 878 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00395579 s, 265 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00179504 s, 584 MB/s 1048576 bytes (1.0 MB) copied, 0.00675198 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00115664 s, 907 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00572088 s, 183 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0899987 s, 186 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00919791 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137357 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173629 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179346 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129949 s, 80.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016718 s, 62.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0184872 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204592 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280836 s, 37.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112272 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129643 s, 80.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289274 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029693 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449074 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0575223 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0585012 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00210637 s, 498 MB/s 1048576 bytes (1.0 MB) copied, 0.0129616 s, 80.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00928689 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111186 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00120301 s, 872 MB/s , 0.00158094 s, 663 MB/s , 0.00554804 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00583362 s, 180 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00141728 s, 740 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151033 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147108 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0163685 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150543 s, 69.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153639 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137781 s, 76.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0186696 s, 56.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 06:22:11 (1721384531) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.28@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.133113 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132279 s, 79.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0132493 s, 79.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0253487 s, 41.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228048 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0266384 s, 39.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0439434 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0402227 s, 26.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0450313 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0445908 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0438819 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0458086 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421892 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0460147 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366522 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455148 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0458703 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00851694 s, 123 MB/s 1+0 records in 1+0 records out , 0.00921058 s, 114 MB/s , 0.0048148 s, 218 MB/s 1048576 bytes (1.0 MB) copied, 0.0102746 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0099879 s, 105 MB/s , 0.00946287 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00931521 s, 113 MB/s 1048576 bytes (1.0 MB) copied, 0.00730235 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00891591 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00886999 s, 118 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00896092 s, 117 MB/s , 0.0126776 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139872 s, 75.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0142838 s, 73.4 MB/s , 0.0135742 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156445 s, 67.0 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.104717 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899872 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109359 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215884 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250364 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221816 s, 47.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271448 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0353095 s, 29.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0363499 s, 28.8 MB/s , 0.0382599 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028281 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388416 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388809 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0320795 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373856 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.033608 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338713 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0113729 s, 92.2 MB/s , 0.00961412 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893501 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00874487 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101295 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116993 s, 89.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103195 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115016 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011792 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110128 s, 95.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011253 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112481 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114162 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123399 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122421 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126193 s, 83.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0950939 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00795993 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010689 s, 98.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00643837 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012181 s, 86.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00693932 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00828586 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102899 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115886 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140604 s, 74.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0346558 s, 30.3 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0269713 s, 38.9 MB/s , 0.0471487 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410316 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026187 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467749 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0525936 s, 19.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=7 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108871 s, 96.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00814221 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00792953 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948642 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00816851 s, 128 MB/s 1048576 bytes (1.0 MB) copied, 0.0120037 s, 87.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0131397 s, 79.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096518 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143909 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102427 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101745 s, 103 MB/s 1048576 bytes (1.0 MB) copied, 0.0143933 s, 72.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0119239 s, 87.9 MB/s , 0.0115683 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121405 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126662 s, 82.8 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.121363 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101909 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900159 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0235476 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01418 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023605 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.041771 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333733 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.034917 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279214 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237582 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338211 s, 31.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.039806 s, 26.3 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0471723 s, 22.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=5 , 0.0434403 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537126 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0337165 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00739229 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00683909 s, 153 MB/s , 0.0100132 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053291 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00820692 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898924 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00141787 s, 740 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00842261 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00165601 s, 633 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0095907 s, 109 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00951224 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00355973 s, 295 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00134258 s, 781 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110619 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115531 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112915 s, 92.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (6s) == sanityn test 77f: check TBF JobID nrs policy ========== 06:22:17 (1721384537) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.139271 s, 120 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00826215 s, 127 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164309 s, 63.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0204168 s, 51.4 MB/s , 0.0101659 s, 103 MB/s 1048576 bytes (1.0 MB) copied, 0.0184575 s, 56.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125432 s, 83.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00801646 s, 131 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101201 s, 104 MB/s 1048576 bytes (1.0 MB) copied, 0.0313649 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296192 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375618 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0397367 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038277 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0248192 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385611 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366645 s, 28.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00919058 s, 114 MB/s , 0.00743332 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00871199 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116149 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803414 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00981442 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00856376 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00864899 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0095168 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010166 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107096 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01083 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130098 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112471 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118327 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136882 s, 76.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0856983 s, 196 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115423 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00719416 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183891 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00848017 s, 124 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0167461 s, 62.6 MB/s , 0.0192419 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237249 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138006 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122845 s, 85.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0487534 s, 21.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0430734 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452256 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0523007 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.056323 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0569065 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394567 s, 26.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00618618 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00853522 s, 123 MB/s , 0.00736642 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00621092 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0088337 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0081714 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123017 s, 85.2 MB/s , 0.00972818 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00905225 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00978949 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102226 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105458 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114811 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111396 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914084 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120222 s, 87.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.128729 s, 130 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00730231 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017048 s, 61.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177471 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.027975 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270471 s, 38.8 MB/s , 0.0333602 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0288769 s, 36.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300434 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0249935 s, 42.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0224261 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185355 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0225288 s, 46.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0371303 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250307 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296047 s, 35.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228657 s, 45.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00640781 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00879623 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00474135 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00129413 s, 810 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00834376 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0011242 s, 933 MB/s , 0.00846907 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00121604 s, 862 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00942114 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00690794 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0014069 s, 745 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00139966 s, 749 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109581 s, 95.7 MB/s , 0.0102496 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103942 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00294507 s, 356 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0920233 s, 182 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00936882 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014863 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100691 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205469 s, 51.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0299161 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119103 s, 88.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0346372 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028771 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0355369 s, 29.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0270832 s, 38.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0239198 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137866 s, 76.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0278563 s, 37.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017124 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274375 s, 38.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159214 s, 65.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00921193 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00854699 s, 123 MB/s 1048576 bytes (1.0 MB) copied, 0.00857058 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00174122 s, 602 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00241798 s, 434 MB/s 1048576 bytes (1.0 MB) copied, 0.00703547 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127569 s, 82.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00646746 s, 162 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00464615 s, 226 MB/s , 0.0121323 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115897 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102758 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119868 s, 87.5 MB/s , 0.012524 s, 83.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0038712 s, 271 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173304 s, 60.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (6s) == sanityn test 77g: Change TBF type directly ============ 06:22:23 (1721384543) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (2s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 06:22:25 (1721384545) oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory pdsh@oleg328-client: oleg328-server: ssh exited with exit code 2 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 12 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 12 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 pdsh@oleg328-client: oleg328-server: ssh exited with exit code 12 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 06:22:28 (1721384548) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 06:22:34 (1721384554) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.082435 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00755549 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00838504 s, 125 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134527 s, 77.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0123895 s, 84.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.028646 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030289 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273952 s, 38.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0185277 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218418 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251823 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250887 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0684237 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0781101 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0740023 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0633633 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0697524 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00866111 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00886313 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00844312 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010478 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122759 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01036 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109323 s, 95.9 MB/s , 0.0107013 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113227 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122737 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123164 s, 85.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114971 s, 91.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013391 s, 78.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0119651 s, 87.6 MB/s , 0.0140026 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144074 s, 72.8 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93987 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4296 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (50s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 06:23:24 (1721384604) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.640336 s, 26.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00919497 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118111 s, 88.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259002 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0278939 s, 37.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0350872 s, 29.9 MB/s , 0.0268689 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375949 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329672 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.371651 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.381685 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.393798 s, 2.7 MB/s , 0.389399 s, 2.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.386786 s, 2.7 MB/s , 0.398636 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.390495 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.404257 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.147762 s, 7.1 MB/s , 0.14977 s, 7.0 MB/s , 0.147875 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.148842 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.148492 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.149198 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.148766 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.14979 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.15126 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151226 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.150631 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151573 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.154228 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152183 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151951 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153928 s, 6.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8326 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9663 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.656644 s, 25.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153899 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.18288 s, 5.7 MB/s , 0.184772 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.170206 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.192124 s, 5.5 MB/s 1048576 bytes (1.0 MB) copied, 0.182209 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.188305 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.182549 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.353243 s, 3.0 MB/s , 0.348573 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.391151 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.38081 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.37937 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.397814 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.386502 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.393044 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.132875 s, 7.9 MB/s , 0.130858 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.131332 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135024 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13196 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.133016 s, 7.9 MB/s , 0.13292 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.133649 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134118 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134258 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136184 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135506 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.135357 s, 7.7 MB/s , 0.135083 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135407 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.138184 s, 7.6 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8409 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9578 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (134s) == sanityn test 77k: check TBF policy with NID/JobID/OPCode expression ========================================================== 06:25:38 (1721384738) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.199152 s, 84.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0139058 s, 75.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126868 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023134 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172713 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0304529 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0244969 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197486 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226429 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0633518 s, 16.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0576715 s, 18.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0784974 s, 13.4 MB/s , 0.0635965 s, 16.5 MB/s , 0.0602267 s, 17.4 MB/s , 0.0797667 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0607821 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0632547 s, 16.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0018718 s, 560 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00854823 s, 123 MB/s , 0.00887464 s, 118 MB/s , 0.00879731 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00881745 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00838243 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011649 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00897343 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0085723 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00433798 s, 242 MB/s , 0.00369555 s, 284 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102696 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0118848 s, 88.2 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011331 s, 92.5 MB/s , 0.0116205 s, 90.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133143 s, 78.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93427 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73514 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.28@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.28@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.190067 s, 88.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020661 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199179 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0242623 s, 43.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156129 s, 67.2 MB/s 1048576 bytes (1.0 MB) copied, 0.025713 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156794 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00851918 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00923513 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0641271 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0727054 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0835954 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0996311 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.142961 s, 7.3 MB/s , 0.15762 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.247535 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.257316 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0158924 s, 66.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154232 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196485 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151748 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157714 s, 66.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0168295 s, 62.3 MB/s , 0.0162495 s, 64.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179741 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0189364 s, 55.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0180047 s, 58.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230313 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193527 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200963 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218808 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208389 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237015 s, 44.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94587 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73614 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.28@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.215576 s, 77.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165174 s, 63.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148179 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843137 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109559 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00768373 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148699 s, 70.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104903 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208569 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0631082 s, 16.6 MB/s , 0.0611419 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0868788 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105918 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100915 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0947873 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110485 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.105524 s, 9.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100927 s, 104 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137918 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00964068 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140645 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013006 s, 80.6 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156474 s, 67.0 MB/s , 0.0136184 s, 77.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015685 s, 66.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015394 s, 68.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183393 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181221 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174714 s, 60.0 MB/s , 0.018903 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198331 s, 52.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200143 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019449 s, 53.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94583 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.88416 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.377594 s, 44.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0446193 s, 23.5 MB/s , 0.0475955 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102792 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0992025 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.148144 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151625 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135655 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.143181 s, 7.3 MB/s , 0.154933 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.157601 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.161545 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.186905 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.328084 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.343604 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.438673 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.448736 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128059 s, 81.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148866 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169069 s, 62.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0203417 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116816 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119627 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171833 s, 61.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189538 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143561 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172886 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00231689 s, 453 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152949 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162765 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0169917 s, 61.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0170003 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199414 s, 52.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.84273 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74205 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95383 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74104 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={1000} rate=5 running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.65494 s, 25.6 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113845 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0317814 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0350612 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.037087 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332209 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319469 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0324012 s, 32.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0351171 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.36705 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.379251 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.373319 s, 2.8 MB/s 1048576 bytes (1.0 MB) copied, 0.383429 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.394532 s, 2.7 MB/s 1048576 bytes (1.0 MB) copied, 0.389234 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.39521 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.400569 s, 2.6 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00698684 s, 150 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104659 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717525 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00821521 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00425625 s, 246 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112593 s, 93.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00155199 s, 676 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115774 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108064 s, 97.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010897 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00919848 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0098283 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106778 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118591 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011747 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106403 s, 98.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.6977 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4382 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.193432 s, 86.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00780203 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133324 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147006 s, 71.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205941 s, 50.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00841492 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192437 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172485 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00906568 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0690762 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0882629 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0776801 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0801505 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0866803 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838401 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101446 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0848744 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00671354 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00676518 s, 155 MB/s , 0.00774258 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00980147 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00853404 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105043 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122304 s, 85.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00985072 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01171 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0112544 s, 93.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0098729 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.0118624 s, 88.4 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0116392 s, 90.1 MB/s , 0.01166 s, 89.9 MB/s , 0.0116648 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122976 s, 85.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95547 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.73006 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.334704 s, 50.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00801317 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00631824 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00830876 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00635111 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012688 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0161363 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0051163 s, 205 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00632611 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.174651 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.189403 s, 5.5 MB/s , 0.185722 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.198261 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.365142 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.369606 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.378077 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.363637 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00607295 s, 173 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0179085 s, 58.6 MB/s , 0.00623205 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00649378 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154179 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107984 s, 97.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0156389 s, 67.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00835021 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107657 s, 97.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0194294 s, 54.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00999022 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101831 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107795 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118888 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116489 s, 90.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126339 s, 83.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.78537 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72627 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.77302 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72693 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (368s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 06:31:46 (1721385106) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (2s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 06:31:48 (1721385108) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg328-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.58831 s, 292 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg328-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0169 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 06:32:38 (1721385158) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.101078 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114652 s, 91.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164161 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152957 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174116 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302561 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0144104 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0290104 s, 36.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219056 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0299745 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159486 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0259234 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394948 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0311259 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279699 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0285236 s, 36.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012a5e9800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a5e9800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452655 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00232369 s, 451 MB/s , 0.0089681 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101885 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105533 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00986783 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129505 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00483427 s, 217 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100891 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124551 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00572019 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106201 s, 98.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0128534 s, 81.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105006 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120287 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130776 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00172956 s, 606 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94741 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.96585 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.141938 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00900158 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0220329 s, 47.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177005 s, 59.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0187888 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233155 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0292647 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018999 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0239226 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0703217 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0862365 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.076874 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0868902 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.167025 s, 6.3 MB/s , 0.157783 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.271254 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.265176 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166126 s, 63.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174071 s, 60.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179287 s, 58.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200043 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195957 s, 53.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0181074 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0183408 s, 57.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0176113 s, 59.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197665 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187927 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187768 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191141 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0197459 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019917 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200823 s, 52.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219726 s, 47.7 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg328-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg328-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94322 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.96778 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 06:33:44 (1721385224) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (3s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 06:33:47 (1721385227) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (25s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 06:34:12 (1721385252) mds.MDS.mdt.nrs_policies=tbf oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long pdsh@oleg328-client: oleg328-server: ssh exited with exit code 36 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument pdsh@oleg328-client: oleg328-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (10s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 06:34:23 (1721385263) oleg328-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: Resource temporarily unavailable pdsh@oleg328-client: oleg328-server: ssh exited with exit code 11 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 06:34:25 (1721385265) File: '/mnt/lustre/d79.sanityn' Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272505431 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:34:25.000000000 -0400 Modify: 2024-07-19 06:34:25.000000000 -0400 Change: 2024-07-19 06:34:25.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 06:34:37 (1721385277) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 06:34:38 (1721385278) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (0s) == sanityn test 81a: rename and stat under striped directory ========================================================== 06:34:38 (1721385278) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 06:34:39 (1721385279) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 06:34:40 (1721385280) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 06:34:41 (1721385281) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.6859 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.6859 PASS 82 (1s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 06:34:42 (1721385282) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 06:34:43 (1721385283) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 90: open/create and unlink striped directory ========================================================== 06:34:50 (1721385290) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 06:34:51 (1721385291) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 06:34:52 (1721385292) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 06:34:53 (1721385293) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2646 0xa56 0 1 2627 0xa43 0 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2628 0xa44 0 0 2647 0xa57 0 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 06:35:01 (1721385301) osc.lustre-OST0000-osc-ffff8800b5884000.idle_timeout=0 osc.lustre-OST0000-osc-ffff88012a5e9800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800b5884000.idle_timeout=0 osc.lustre-OST0001-osc-ffff88012a5e9800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0370565 s, 221 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 2.99929 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5164: 4985 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0140296 s, 292 kB/s osc.lustre-OST0000-osc-ffff8800b5884000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012a5e9800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5884000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a5e9800.idle_timeout=debug PASS 94 (9s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 06:35:10 (1721385310) llite.lustre-ffff8800b5884000.fast_read=0 llite.lustre-ffff88012a5e9800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00154755 s, 7.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800b5884000.fast_read=1 llite.lustre-ffff88012a5e9800.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 06:35:18 (1721385318) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 06:35:19 (1721385319) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00127657 s, 3.2 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (1s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 06:35:20 (1721385320) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0220104 s, 95.3 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 06:35:22 (1721385322) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0293128 s, 71.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 06:35:24 (1721385324) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00197022 s, 6.1 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 06:35:26 (1721385326) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000570632 s, 7.2 MB/s PASS 101a (1s) == sanityn test 101b: Discard DoM data on rename ========= 06:35:27 (1721385327) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000657479 s, 6.2 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 06:35:29 (1721385329) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000442995 s, 9.2 MB/s PASS 101c (2s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 06:35:32 (1721385332) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1721385332 lockahead_test: lockahead_test.c:1186: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1721385335 Finishing test test23 at 1721385335 PASS 103 (6s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 06:35:38 (1721385338) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 06:35:39 (1721385339) fail_loc=0x416 PASS 105 (47s) == sanityn test 106a: Verify the btime via statx() ======= 06:36:26 (1721385386) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 06:36:27 (1721385387) SKIP: sanityn test_106b statx() only test SKIP 106b (1s) == sanityn test 106c: Verify statx attributes mask ======= 06:36:28 (1721385388) SKIP: sanityn test_106c statx() only test SKIP 106c (0s) == sanityn test 107a: Basic grouplock conflict =========== 06:36:28 (1721385388) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0616795 s, 170 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2633 0xa49 0 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 06:36:32 (1721385392) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.059436 s, 176 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2653 0xa5d 0 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6859 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 06:36:40 (1721385400) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.00157248 s, 6.4 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.23051 s, 2.0 MB/s PASS 108a (6s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 06:36:46 (1721385406) 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 5 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 9 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 19 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 21 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 35 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions oleg328-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 41 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 192.168.203.128@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.128@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg328-client.virtnet /mnt/lustre (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg328-server' oleg328-server: oleg328-server.virtnet: executing load_modules_local oleg328-server: Loading modules from /home/green/git/lustre-release/lustre oleg328-server: detected 4 online CPUs by sysfs oleg328-server: Force libcfs to create 2 CPU partitions Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre Starting client: oleg328-client.virtnet: -o user_xattr,flock oleg328-server@tcp:/lustre /mnt/lustre2 PASS 109 (391s) == sanityn test 110: do not grant another lock on resend ========================================================== 06:43:17 (1721385797) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 06:43:17.000000000 -0400 Modify: 2024-07-19 06:43:17.000000000 -0400 Change: 2024-07-19 06:43:17.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg328-client.virtnet oleg328-client.virtnet PASS 110 (48s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 06:44:05 (1721385845) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 06:44:06 (1721385846) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 06:44:07 (1721385847) lustre-OST0000-osc-ffff8800a1ebd000 active. lustre-OST0001-osc-ffff8800a1ebd000 active. lustre-OST0000-osc-ffff8800a1ebd000 active. lustre-OST0001-osc-ffff8800a1ebd000 active. MGC192.168.203.128@tcp active. PASS 113 (2s) cleanup: ====================================================== == sanityn test complete, duration 3985 sec ============== 06:44:09 (1721385849) Stopping clients: oleg328-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg328-client.virtnet /mnt/lustre2 opts: