-----============= acceptance-small: sanityn ============----- Fri Jul 19 12:19:29 EDT 2024 excepting tests: 27 28 102 skipping tests SLOW=no: 33a Starting client oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Started clients oleg340-client.virtnet: 192.168.203.140@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) oleg340-client.virtnet: executing check_config_client /mnt/lustre oleg340-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg340-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9cd2800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6e50800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9cd2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6e50800.idle_timeout=debug disable quota as required oleg340-server: oleg340-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164119 s, 63.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6837] == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 12:19:39 (1721405979) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (2s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 12:19:41 (1721405981) -rw-r--r-- 1 root root 0 Jul 19 12:19 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (1s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 12:19:42 (1721405982) -rw-r--r-- 1 root root 0 Jul 19 12:19 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (1s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 12:19:44 (1721405984) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 12:19:45 (1721405985) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (2s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 12:19:47 (1721405987) total 1 -rwxrwxrwx 1 root root 0 Jul 19 12:19 f2a -rwxrwxrwx 1 root root 0 Jul 19 12:19 f2b total 1 -rwxrwxrwx 1 root root 0 Jul 19 12:19 f2a -rwxrwxrwx 1 root root 0 Jul 19 12:19 f2b running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00409022 s, 125 kB/s PASS 2e (1s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 12:19:48 (1721405988) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 12:19:49 (1721405989) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0959824 s, 21.8 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (2s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 12:19:51 (1721405991) PASS 3 (1s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 12:19:52 (1721405992) PASS 4 (3s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 12:19:55 (1721405995) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 12:19:57 (1721405997) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (1s) == sanityn test 7: remove of open directory on other node ======================================================================= 12:19:58 (1721405998) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 12:20:00 (1721406000) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (1s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 12:20:01 (1721406001) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 12:20:03 (1721406003) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109379 s, 95.9 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.0120102 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (2s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 12:20:05 (1721406005) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00147233 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00144463 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00143905 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00146641 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0016128 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00145896 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00156954 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00195644 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00142584 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00149482 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00155566 s, 0.6 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00156207 s, 0.6 kB/s PASS 10a (1s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 12:20:07 (1721406007) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00223866 s, 1.4 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00659465 s, 621 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000316667 s, 9.7 MB/s PASS 10b (1s) == sanityn test 11: execution of file opened for write should return error ============================================================== 12:20:08 (1721406008) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 273: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 12:20:10 (1721406010) warning: '-runtime' deprecated, use '-t runtime' instead - link 2957 (time 1721406020.50 total 10.00 last 295.62) - link 6132 (time 1721406030.50 total 20.00 last 317.48) - link 9181 (time 1721406040.51 total 30.01 last 304.80) - link 10000 (time 1721406043.32 total 32.81 last 291.61) - link 13254 (time 1721406053.32 total 42.82 last 325.38) - link 16790 (time 1721406063.32 total 52.82 last 353.57) - link 19858 (time 1721406073.32 total 62.82 last 306.77) - link 23086 (time 1721406083.32 total 72.82 last 322.72) - link 26295 (time 1721406093.32 total 82.82 last 320.84) - link 29572 (time 1721406103.32 total 92.82 last 327.68) total: 31939 link in 100.00 seconds: 319.39 ops/second using seed 1084725248 running for 100 seconds - stat 10000 (time 1721406014 ; total 3 ; last 3) - stat 20000 (time 1721406017 ; total 6 ; last 3) - stat 30000 (time 1721406020 ; total 9 ; last 3) - stat 40000 (time 1721406023 ; total 12 ; last 3) - stat 50000 (time 1721406025 ; total 14 ; last 2) - stat 60000 (time 1721406028 ; total 17 ; last 3) - stat 70000 (time 1721406031 ; total 20 ; last 3) - stat 80000 (time 1721406034 ; total 23 ; last 3) - stat 90000 (time 1721406037 ; total 26 ; last 3) - stat 100000 (time 1721406040 ; total 29 ; last 3) - stat 110000 (time 1721406043 ; total 32 ; last 3) - stat 120000 (time 1721406045 ; total 34 ; last 2) - stat 130000 (time 1721406048 ; total 37 ; last 3) - stat 140000 (time 1721406051 ; total 40 ; last 3) - stat 150000 (time 1721406054 ; total 43 ; last 3) - stat 160000 (time 1721406056 ; total 45 ; last 2) - stat 170000 (time 1721406059 ; total 48 ; last 3) - stat 180000 (time 1721406062 ; total 51 ; last 3) - stat 190000 (time 1721406067 ; total 56 ; last 5) - stat 200000 (time 1721406073 ; total 62 ; last 6) - stat 210000 (time 1721406078 ; total 67 ; last 5) - stat 220000 (time 1721406083 ; total 72 ; last 5) - stat 230000 (time 1721406089 ; total 78 ; last 6) - stat 240000 (time 1721406094 ; total 83 ; last 5) - stat 250000 (time 1721406099 ; total 88 ; last 5) - stat 260000 (time 1721406104 ; total 93 ; last 5) - stat 270000 (time 1721406110 ; total 99 ; last 6) total: 272269 stats in 100 seconds: 2722.689941 stats/second - unlinked 0 (time 1721406061 ; total 0 ; last 0) - unlinked 10000 (time 1721406109 ; total 48 ; last 48) - unlinked 20000 (time 1721406130 ; total 69 ; last 21) - unlinked 30000 (time 1721406152 ; total 91 ; last 22) unlink(/mnt/lustre2/lockdir/lockfile31939) error: No such file or directory total: 31938 unlinks in 95 seconds: 336.189484 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (20431) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (20434) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 12:22:37 (1721406157) f13 f13 PASS 13 (2s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 12:22:39 (1721406159) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 308: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4253: 21814 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (1s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 12:22:40 (1721406160) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4253: 22441 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 12:22:42 (1721406162) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 323: 23067 Terminated $DIR1/$tdir/sleep 60 PASS 14b (1s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 12:22:43 (1721406163) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: 23696 Terminated $DIR1/$tdir/sleep 60 PASS 14c (1s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 12:22:44 (1721406164) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 346: 24324 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 12:22:46 (1721406166) PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: STRIPECOUNT=2 ORIGFREE=7517184 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7517184kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed sleep 5 for ZFS zfs Waiting for MDT destroys to complete SKIP 15 (10s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 12:22:56 (1721406176) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0859455 s, 24.4 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0579175 s, 36.2 MB/s Chance of close/open is 1 in 50 Seed set to 6177 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x30d22e 000008[0] 1721406177.863602 trunc from 00000000 to 0x30d22d (0x30d22e bytes) 000011[0] 1721406177.875381 read 0x1ebb12 thru 0x1f9cb3 (0xe1a2 bytes) 000014[1] 1721406177.884612 mapread 0x05944b thru 0x068f66 (0xfb1c bytes) 000017[1] 1721406177.900824 mapread 0x16b2cb thru 0x1772b9 (0xbfef bytes) 000018[0] 1721406177.909896 write 0x000048 thru 0x0009ed (0x9a6 bytes) 000020[1] 1721406177.916367 read 0x00af9e thru 0x019e4d (0xeeb0 bytes) 000022[0] 1721406177.922469 write 0x2cf9fd thru 0x2d5f97 (0x659b bytes) truncating to largest ever: 0x4d00c8 000023[0] 1721406177.926339 trunc from 0x30d22e to 0x4d00c7 (0x1c2e9a bytes) 000024[0] 1721406177.935122 mapread 0x174cef thru 0x184176 (0xf488 bytes) 000027[1] 1721406177.951516 write 0x2f7d68 thru 0x2fc552 (0x47eb bytes) truncating to largest ever: 0x71803e 000028[0] 1721406177.962019 trunc from 0x4d00c8 to 0x71803d (0x247f76 bytes) 000030[0] 1721406177.975606 trunc from 0x71803e to 0x0870b1 (0x690f8c bytes) 000032[1] 1721406177.986121 mapread 0x054ae3 thru 0x05f774 (0xac92 bytes) 000034[1] 1721406177.997173 mapwrite 0x5eb394 thru 0x5f00d0 (0x4d3d bytes) 000036[1] 1721406178.034733 trunc from 0x5f00d1 to 0x268e32 (0x38729e bytes) 000038[0] 1721406178.042941 write 0x81bc22 thru 0x82399b (0x7d7a bytes) 000041[1] 1721406178.056125 read 0x36332d thru 0x3709bb (0xd68f bytes) 000042[1] 1721406178.074544 trunc from 0x82399c to 0x1c76db (0x65c2c0 bytes) 000044[1] 1721406178.082569 mapread 0x034898 thru 0x04415d (0xf8c6 bytes) 000047[1] 1721406178.107063 mapread 0x096c21 thru 0x0a5fa4 (0xf384 bytes) 000048[0] 1721406178.126444 mapread 0x1bb978 thru 0x1c39e6 (0x806f bytes) 000053[0] 1721406178.144891 mapwrite 0x4b169c thru 0x4b1976 (0x2db bytes) 000054[0] 1721406178.168182 write 0x11cd35 thru 0x120e7f (0x414b bytes) 000056[0] 1721406178.176825 trunc from 0x4b1977 to 0x1c1799 (0x2f01dd bytes) 000060[0] 1721406178.191858 mapwrite 0x0191e2 thru 0x0246d7 (0xb4f6 bytes) 000061[0] 1721406178.211318 write 0x38988b thru 0x398520 (0xec96 bytes) 000064[1] 1721406178.219453 read 0x30349d thru 0x312e99 (0xf9fd bytes) 000068[1] 1721406178.230708 mapread 0x1a62ae thru 0x1a8df2 (0x2b45 bytes) 000069[1] 1721406178.238478 trunc from 0x398521 to 0x2f4cff (0xa3821 bytes) 000070[0] 1721406178.246840 mapread 0x15ad69 thru 0x169a7c (0xed14 bytes) 000071[0] 1721406178.257125 trunc from 0x2f4d00 to 0x6e01f2 (0x3eb4f3 bytes) 000073[1] 1721406178.265346 trunc from 0x6e01f3 to 0x00c60d (0x6d3be5 bytes) 000074[0] 1721406178.272322 write 0x9c5d8e thru 0x9ced93 (0x9006 bytes) 000076[1] 1721406178.283545 mapread 0x7b8fcf thru 0x7be648 (0x567a bytes) 000077[1] 1721406178.307481 mapwrite 0x05dc07 thru 0x068324 (0xa71e bytes) 000079[1] 1721406178.332406 trunc from 0x9ced94 to 0x2831e2 (0x74bbb1 bytes) 000080[0] 1721406178.348246 write 0x48eec3 thru 0x496708 (0x7846 bytes) 000081[1] 1721406178.355072 read 0x0f7512 thru 0x101de3 (0xa8d2 bytes) 000082[0] 1721406178.375871 mapwrite 0x79fdd4 thru 0x7ad010 (0xd23d bytes) truncating to largest ever: 0x7fa8f5 000083[1] 1721406178.406926 trunc from 0x7ad011 to 0x7fa8f4 (0x4d8e4 bytes) 000085[1] 1721406178.419982 mapwrite 0x845965 thru 0x8471be (0x185a bytes) 000087[0] 1721406178.448598 mapread 0x5c3f7b thru 0x5c7f7a (0x4000 bytes) 000090[1] 1721406178.466201 mapread 0x332b13 thru 0x33cc90 (0xa17e bytes) 000091[1] 1721406178.480749 mapwrite 0x70f354 thru 0x71c538 (0xd1e5 bytes) 000092[1] 1721406178.511724 read 0x0b9e81 thru 0x0c17b1 (0x7931 bytes) 000093[1] 1721406178.534271 mapwrite 0x2f539b thru 0x301c41 (0xc8a7 bytes) 000094[0] 1721406178.551925 write 0x584dad thru 0x593687 (0xe8db bytes) 000096[0] 1721406178.566818 mapread 0x67b6dd thru 0x6846b2 (0x8fd6 bytes) 000097[1] 1721406178.576826 mapwrite 0x689e11 thru 0x6983dc (0xe5cc bytes) 000099[1] 1721406178.616242 write 0x6ee0ef thru 0x6fba37 (0xd949 bytes) 000100[0] 1721406178.621998 read 0x74e7fc thru 0x751e35 (0x363a bytes) 000102[1] 1721406178.633082 read 0x0e3c22 thru 0x0ee877 (0xac56 bytes) 000103[1] 1721406178.638677 trunc from 0x8471bf to 0x5ca6e9 (0x27cad5 bytes) 000104[0] 1721406178.651602 trunc from 0x5ca6ea to 0x70256a (0x137e81 bytes) 000106[1] 1721406178.659198 mapwrite 0x91b70e thru 0x924010 (0x8903 bytes) 000107[0] 1721406178.709307 mapread 0x0d39ab thru 0x0dc8f3 (0x8f49 bytes) 000109[1] 1721406178.735336 trunc from 0x924011 to 0x55b682 (0x3c898e bytes) 000111[0] 1721406178.747123 trunc from 0x55b683 to 0x4a4c77 (0xb6a0b bytes) 000112[1] 1721406178.756164 read 0x284ba4 thru 0x287123 (0x2580 bytes) 000113[0] 1721406178.762852 read 0x1be01b thru 0x1c3192 (0x5178 bytes) 000118[0] 1721406178.786803 mapwrite 0x62d224 thru 0x638c8c (0xba69 bytes) 000119[0] 1721406178.816461 write 0x3aa099 thru 0x3b7288 (0xd1f0 bytes) 000120[0] 1721406178.817102 write 0x4dc505 thru 0x4df3b5 (0x2eb1 bytes) 000122[0] 1721406178.817670 mapread 0x00d916 thru 0x011021 (0x370c bytes) 000127[0] 1721406178.833402 trunc from 0x638c8d to 0x69e795 (0x65b09 bytes) 000128[0] 1721406178.837606 read 0x647b9b thru 0x6562f9 (0xe75f bytes) 000129[0] 1721406178.841421 trunc from 0x69e796 to 0x680e4e (0x1d947 bytes) 000130[0] 1721406178.844638 mapread 0x49cd62 thru 0x4a43fd (0x769c bytes) 000132[1] 1721406178.857821 mapwrite 0x61523b thru 0x61d416 (0x81dc bytes) 000133[1] 1721406178.889260 read 0x2fdf91 thru 0x30276c (0x47dc bytes) 000134[0] 1721406178.913237 read 0x21e616 thru 0x21fd7b (0x1766 bytes) 000137[0] 1721406178.934066 write 0x6780da thru 0x678b0a (0xa31 bytes) 000138[1] 1721406178.939734 read 0x05359c thru 0x0578b1 (0x4316 bytes) 000139[1] 1721406178.958973 mapread 0x11c940 thru 0x1270e9 (0xa7aa bytes) 000140[1] 1721406178.959346 trunc from 0x680e4f to 0x255e46 (0x42b008 bytes) 000141[1] 1721406178.974692 mapread 0x141da4 thru 0x147d99 (0x5ff6 bytes) 000142[1] 1721406178.982680 read 0x0b768e thru 0x0c30e9 (0xba5c bytes) 000143[1] 1721406178.989413 mapwrite 0x6aaf89 thru 0x6ba164 (0xf1dc bytes) 000144[0] 1721406179.026145 write 0x4d819a thru 0x4d92cb (0x1132 bytes) 000145[1] 1721406179.038204 mapwrite 0x456d09 thru 0x45cbc6 (0x5ebe bytes) 000146[1] 1721406179.069424 read 0x61ced7 thru 0x627e40 (0xaf6a bytes) 000147[0] 1721406179.074366 write 0x686149 thru 0x6891d5 (0x308d bytes) 000148[0] 1721406179.081881 mapwrite 0x1f7a6e thru 0x1fba6e (0x4001 bytes) 000149[1] 1721406179.106046 mapwrite 0x11945f thru 0x11f27f (0x5e21 bytes) 000150[1] 1721406179.133191 mapwrite 0x27a085 thru 0x284daf (0xad2b bytes) 000152[0] 1721406179.163013 mapread 0x4d285f thru 0x4da25a (0x79fc bytes) 000153[0] 1721406179.178806 trunc from 0x6ba165 to 0x3fc53d (0x2bdc27 bytes) 000154[0] 1721406179.190923 mapread 0x1d1410 thru 0x1dea49 (0xd63a bytes) 000156[1] 1721406179.211403 write 0x8c8699 thru 0x8d30ec (0xaa54 bytes) 000157[0] 1721406179.218212 write 0x2865d5 thru 0x288ae0 (0x250c bytes) 000162[1] 1721406179.240435 mapread 0x244ad4 thru 0x24ea81 (0x9fae bytes) 000166[1] 1721406179.266556 read 0x75d08e thru 0x767bc0 (0xab33 bytes) 000167[0] 1721406179.275232 read 0x099570 thru 0x0a251a (0x8fab bytes) 000169[1] 1721406179.298674 read 0x65f57d thru 0x667dc3 (0x8847 bytes) 000170[0] 1721406179.303460 mapwrite 0x46ffff thru 0x473b63 (0x3b65 bytes) 000172[1] 1721406179.343482 mapread 0x6d2133 thru 0x6dc2b1 (0xa17f bytes) 000173[1] 1721406179.358129 trunc from 0x8d30ed to 0x7e03fb (0xf2cf1 bytes) 000174[1] 1721406179.372966 read 0x24678c thru 0x24d4e9 (0x6d5e bytes) 000175[1] 1721406179.392107 read 0x4c9b7f thru 0x4d0c09 (0x708b bytes) 000176[1] 1721406179.395714 mapread 0x3aab9e thru 0x3b8678 (0xdadb bytes) 000177[1] 1721406179.396199 mapread 0x213463 thru 0x218872 (0x5410 bytes) 000178[0] 1721406179.405243 mapread 0x3fbeb8 thru 0x40bc58 (0xfda1 bytes) 000182[0] 1721406179.433431 mapwrite 0x7d2197 thru 0x7d87c6 (0x6630 bytes) 000183[0] 1721406179.458385 mapread 0x45ae5d thru 0x46288d (0x7a31 bytes) 000184[1] 1721406179.468924 write 0x37ea09 thru 0x37f27e (0x876 bytes) 000186[1] 1721406179.477967 mapread 0x62398b thru 0x62d69d (0x9d13 bytes) 000187[1] 1721406179.488793 read 0x6c7dc7 thru 0x6d524a (0xd484 bytes) 000188[0] 1721406179.496613 read 0x2426d4 thru 0x24a3f5 (0x7d22 bytes) 000190[0] 1721406179.505609 trunc from 0x7e03fc to 0x57b2d6 (0x265125 bytes) 000191[1] 1721406179.519635 trunc from 0x57b2d7 to 0x47442f (0x106ea7 bytes) 000192[1] 1721406179.526426 read 0x2de0ce thru 0x2e29b7 (0x48ea bytes) 000193[0] 1721406179.538394 write 0x2dabdb thru 0x2e87b3 (0xdbd9 bytes) 000195[1] 1721406179.550231 mapwrite 0x940d53 thru 0x9457c8 (0x4a76 bytes) 000196[0] 1721406179.588881 write 0x08d918 thru 0x098bdc (0xb2c5 bytes) 000197[0] 1721406179.598381 mapwrite 0x0e7c0f thru 0x0e91d5 (0x15c7 bytes) 000198[0] 1721406179.619897 mapread 0x0ce7dd thru 0x0d6c93 (0x84b7 bytes) 000199[0] 1721406179.635329 read 0x8aecba thru 0x8b8c49 (0x9f90 bytes) 000200[1] 1721406179.641842 mapwrite 0x09d7b8 thru 0x0a30a0 (0x58e9 bytes) 000205[1] 1721406179.679956 mapwrite 0x915be6 thru 0x92484c (0xec67 bytes) 000207[0] 1721406179.697687 mapread 0x082e1e thru 0x08746e (0x4651 bytes) 000208[1] 1721406179.707914 mapread 0x0d966f thru 0x0e3ff7 (0xa989 bytes) 000209[0] 1721406179.718649 mapwrite 0x5a457c thru 0x5b09a5 (0xc42a bytes) 000211[0] 1721406179.757315 write 0x42e53c thru 0x437d9f (0x9864 bytes) 000213[1] 1721406179.765242 mapwrite 0x83fdf2 thru 0x84caef (0xccfe bytes) 000214[0] 1721406179.802009 mapwrite 0x36ee0d thru 0x37c94c (0xdb40 bytes) 000215[0] 1721406179.834916 mapread 0x84229a thru 0x84d120 (0xae87 bytes) 000216[0] 1721406179.851464 read 0x87cc7e thru 0x888bb0 (0xbf33 bytes) 000217[0] 1721406179.855714 write 0x1da2cb thru 0x1deb22 (0x4858 bytes) 000218[0] 1721406179.858238 write 0x93348d thru 0x93463b (0x11af bytes) 000219[0] 1721406179.858719 mapwrite 0x0e9702 thru 0x0ef350 (0x5c4f bytes) 000222[0] 1721406179.888848 mapread 0x2f5fb9 thru 0x2f7b05 (0x1b4d bytes) 000223[1] 1721406179.892110 write 0x8a6283 thru 0x8a8fd8 (0x2d56 bytes) 000224[1] 1721406179.899005 mapwrite 0x8cf03a thru 0x8d53c0 (0x6387 bytes) 000228[0] 1721406179.933549 mapread 0x32774f thru 0x328629 (0xedb bytes) 000229[0] 1721406179.940448 mapwrite 0x0f7527 thru 0x0fbae3 (0x45bd bytes) 000230[0] 1721406179.962070 mapread 0x8ba660 thru 0x8c9867 (0xf208 bytes) 000232[0] 1721406179.983041 trunc from 0x9457c9 to 0x08e549 (0x8b727f bytes) 000236[0] 1721406179.988709 mapread 0x01e5fa thru 0x021fe4 (0x39eb bytes) 000237[1] 1721406179.997497 write 0x407a13 thru 0x4126a9 (0xac97 bytes) 000238[1] 1721406180.003708 write 0x02b373 thru 0x030c8d (0x591b bytes) 000239[0] 1721406180.009179 read 0x34f308 thru 0x352136 (0x2e2f bytes) 000240[1] 1721406180.017757 mapread 0x0219b8 thru 0x0239c1 (0x200a bytes) 000242[0] 1721406180.029448 mapwrite 0x44f52a thru 0x45166a (0x2141 bytes) 000243[1] 1721406180.055478 write 0x61fa46 thru 0x62e317 (0xe8d2 bytes) 000247[0] 1721406180.071614 mapread 0x36f523 thru 0x372725 (0x3203 bytes) 000248[0] 1721406180.076359 write 0x543009 thru 0x54933c (0x6334 bytes) 000249[0] 1721406180.079937 read 0x61d543 thru 0x626565 (0x9023 bytes) 000250[0] 1721406180.090196 write 0x9095c0 thru 0x90fb0d (0x654e bytes) 000251[1] 1721406180.094528 mapread 0x7b910e thru 0x7bf38a (0x627d bytes) 000252[1] 1721406180.107862 write 0x1ff6ad thru 0x202ce6 (0x363a bytes) 000253[0] 1721406180.116656 read 0x58e02d thru 0x5997df (0xb7b3 bytes) 000255[0] 1721406180.127988 read 0x4158b5 thru 0x422723 (0xce6f bytes) 000257[1] 1721406180.146218 mapread 0x7bf78e thru 0x7c40e2 (0x4955 bytes) 000262[1] 1721406180.153885 mapread 0x871c69 thru 0x87524c (0x35e4 bytes) 000266[1] 1721406180.159653 trunc from 0x90fb0e to 0x2a27a7 (0x66d366 bytes) 000267[1] 1721406180.175603 mapwrite 0x125824 thru 0x1309d7 (0xb1b4 bytes) 000268[0] 1721406180.201298 trunc from 0x2a27a8 to 0x4610d3 (0x1be92c bytes) 000271[1] 1721406180.220530 write 0x10fdbc thru 0x115b2a (0x5d6f bytes) 000273[1] 1721406180.231686 read 0x4508f4 thru 0x45d197 (0xc8a4 bytes) 000274[1] 1721406180.240438 read 0x09adfa thru 0x0a3da2 (0x8fa9 bytes) 000275[1] 1721406180.254486 write 0x03e76f thru 0x03f97d (0x120f bytes) 000279[0] 1721406180.261140 mapread 0x2d3bfe thru 0x2e13d5 (0xd7d8 bytes) 000280[0] 1721406180.283090 write 0x72876f thru 0x73593e (0xd1d0 bytes) 000281[0] 1721406180.293117 mapwrite 0x3f9213 thru 0x405136 (0xbf24 bytes) 000282[1] 1721406180.327002 mapread 0x6206fc thru 0x6254e9 (0x4dee bytes) 000284[0] 1721406180.352701 read 0x288416 thru 0x291a4a (0x9635 bytes) 000285[0] 1721406180.365818 mapwrite 0x51e8df thru 0x52cf7f (0xe6a1 bytes) 000286[0] 1721406180.380784 read 0x4456d2 thru 0x453499 (0xddc8 bytes) 000287[1] 1721406180.384546 read 0x4841eb thru 0x48c56c (0x8382 bytes) 000289[1] 1721406180.399674 mapwrite 0x4ee11e thru 0x4ef682 (0x1565 bytes) 000294[0] 1721406180.445543 trunc from 0x73593f to 0x352c60 (0x3e2cde bytes) 000295[1] 1721406180.457005 trunc from 0x352c61 to 0x2bc18d (0x96ad3 bytes) 000296[1] 1721406180.466031 read 0x1c954e thru 0x1cf9d5 (0x6488 bytes) 000298[1] 1721406180.477305 read 0x0cb3b8 thru 0x0cf28a (0x3ed3 bytes) 000299[1] 1721406180.483201 write 0x1c9d62 thru 0x1d7c30 (0xdecf bytes) 000300[0] 1721406180.483871 trunc from 0x2bc18e to 0x342a95 (0x86908 bytes) 000301[0] 1721406180.499158 mapwrite 0x6419d5 thru 0x6504df (0xeb0b bytes) 000305[0] 1721406180.537148 trunc from 0x6504e0 to 0x737a43 (0xe7564 bytes) 000306[0] 1721406180.543363 mapread 0x2ca828 thru 0x2cd727 (0x2f00 bytes) 000307[0] 1721406180.551066 trunc from 0x737a44 to 0x23b726 (0x4fc31d bytes) 000308[0] 1721406180.555060 mapwrite 0x38e7c0 thru 0x39642c (0x7c6d bytes) 000310[0] 1721406180.584386 mapread 0x330c73 thru 0x33e172 (0xd500 bytes) 000315[1] 1721406180.605037 mapread 0x0306bb thru 0x034dd4 (0x471a bytes) 000317[0] 1721406180.616558 write 0x6ad798 thru 0x6b0273 (0x2adc bytes) 000320[0] 1721406180.630981 mapread 0x069d3d thru 0x0705ca (0x688e bytes) 000324[0] 1721406180.643684 trunc from 0x6b0274 to 0x24cc23 (0x463650 bytes) 000325[0] 1721406180.648444 trunc from 0x24cc24 to 0x046c47 (0x205fdc bytes) 000328[0] 1721406180.656287 trunc from 0x046c48 to 0x5e743e (0x5a07f7 bytes) 000329[0] 1721406180.664448 trunc from 0x5e743f to 0x41325d (0x1d41e1 bytes) 000332[1] 1721406180.672087 mapwrite 0x91bd63 thru 0x91ceb0 (0x114e bytes) 000333[1] 1721406180.720759 mapread 0x66e186 thru 0x66f44b (0x12c6 bytes) 000335[0] 1721406180.722524 read 0x6d66bf thru 0x6e0791 (0xa0d3 bytes) 000337[1] 1721406180.730156 mapread 0x5bebbf thru 0x5ca1e9 (0xb62b bytes) truncating to largest ever: 0x82ca54 000338[0] 1721406180.730722 trunc from 0x91ceb1 to 0x82ca53 (0xf045d bytes) 000341[1] 1721406180.748456 read 0x507bcd thru 0x50c79f (0x4bd3 bytes) 000343[0] 1721406180.763473 trunc from 0x82ca54 to 0x2c80b3 (0x5649a0 bytes) 000346[1] 1721406180.779080 write 0x29d7b2 thru 0x29fa17 (0x2266 bytes) truncating to largest ever: 0x9a4a0e 000348[0] 1721406180.789109 trunc from 0x2c80b4 to 0x9a4a0d (0x6dc95a bytes) 000350[0] 1721406180.801049 mapread 0x2a501d thru 0x2b45a3 (0xf587 bytes) 000351[1] 1721406180.809858 mapwrite 0x0d38d7 thru 0x0d789a (0x3fc4 bytes) 000352[0] 1721406180.840711 write 0x6dc392 thru 0x6deb0b (0x277a bytes) 000354[1] 1721406180.849212 trunc from 0x9a4a0e to 0x1a7a78 (0x7fcf95 bytes) 000357[0] 1721406180.865563 mapwrite 0x48470b thru 0x4931b9 (0xeaaf bytes) 000359[1] 1721406180.899426 read 0x18620f thru 0x18b91d (0x570f bytes) 000360[0] 1721406180.908621 write 0x61cf7e thru 0x627e15 (0xae98 bytes) 000361[0] 1721406180.912475 read 0x10a59d thru 0x10dd06 (0x376a bytes) 000362[0] 1721406180.926775 trunc from 0x627e16 to 0x659f68 (0x32153 bytes) 000363[1] 1721406180.935280 write 0x31f80c thru 0x32da2c (0xe221 bytes) 000364[1] 1721406180.943103 read 0x1ac3ab thru 0x1aef07 (0x2b5d bytes) 000366[0] 1721406180.953615 mapwrite 0x26f5e0 thru 0x2795b3 (0x9fd4 bytes) 000367[1] 1721406180.973746 read 0x11d013 thru 0x120d3c (0x3d2a bytes) 000370[0] 1721406180.987796 trunc from 0x659f69 to 0x9657a3 (0x30b83b bytes) 000378[0] 1721406181.007813 write 0x70a350 thru 0x7198de (0xf58f bytes) 000379[1] 1721406181.011029 write 0x77d0e2 thru 0x78bf5e (0xee7d bytes) 000380[0] 1721406181.020727 mapread 0x4daf36 thru 0x4e5cc7 (0xad92 bytes) 000381[1] 1721406181.026118 mapwrite 0x43b40b thru 0x44a679 (0xf26f bytes) 000382[0] 1721406181.072149 mapread 0x615726 thru 0x6174cf (0x1daa bytes) 000383[0] 1721406181.085467 mapwrite 0x906c80 thru 0x909d48 (0x30c9 bytes) 000384[1] 1721406181.115915 trunc from 0x9657a4 to 0x2a6056 (0x6bf74d bytes) 000385[1] 1721406181.127412 mapwrite 0x74689b thru 0x74877b (0x1ee1 bytes) 000387[0] 1721406181.154397 trunc from 0x74877c to 0x4c383f (0x284f3c bytes) 000388[1] 1721406181.163332 mapread 0x13f55d thru 0x14e0d6 (0xeb7a bytes) 000390[1] 1721406181.188141 write 0x3d4e92 thru 0x3d58e9 (0xa58 bytes) 000391[0] 1721406181.195270 trunc from 0x4c3840 to 0x3666e2 (0x15d15d bytes) 000393[1] 1721406181.207565 trunc from 0x3666e3 to 0x68912f (0x322a4d bytes) 000394[0] 1721406181.217320 read 0x44da4e thru 0x4507da (0x2d8d bytes) 000395[0] 1721406181.227544 write 0x46e11c thru 0x478bd0 (0xaab5 bytes) 000398[1] 1721406181.242694 trunc from 0x689130 to 0x110b99 (0x578596 bytes) 000401[0] 1721406181.259452 write 0x853582 thru 0x8576a7 (0x4126 bytes) 000402[0] 1721406181.266501 read 0x109fa7 thru 0x117c35 (0xdc8f bytes) 000405[0] 1721406181.292694 read 0x60897b thru 0x6133d3 (0xaa59 bytes) 000406[0] 1721406181.293110 write 0x64d469 thru 0x6589bd (0xb555 bytes) 000407[1] 1721406181.293923 write 0x77cf8f thru 0x7809c2 (0x3a34 bytes) 000409[1] 1721406181.306347 mapwrite 0x563cdf thru 0x56688b (0x2bad bytes) 000410[1] 1721406181.328840 mapread 0x63451f thru 0x63c769 (0x824b bytes) 000415[1] 1721406181.350351 mapread 0x8477a5 thru 0x84ad36 (0x3592 bytes) 000418[0] 1721406181.358123 mapwrite 0x74b9c5 thru 0x7577b0 (0xbdec bytes) 000419[0] 1721406181.387145 mapread 0x5a5434 thru 0x5a93c2 (0x3f8f bytes) 000420[0] 1721406181.392621 read 0x5968ca thru 0x598496 (0x1bcd bytes) 000421[0] 1721406181.396510 trunc from 0x8576a8 to 0x830295 (0x27412 bytes) 000422[0] 1721406181.406876 trunc from 0x830296 to 0x000c38 (0x82f65d bytes) 000425[0] 1721406181.416200 write 0x96831e thru 0x97306f (0xad52 bytes) 000428[1] 1721406181.425019 trunc from 0x973070 to 0x5f019f (0x382ed0 bytes) 000429[1] 1721406181.436873 mapwrite 0x73d8e9 thru 0x74b99f (0xe0b7 bytes) 000430[1] 1721406181.474613 trunc from 0x74b9a0 to 0x86537e (0x1199df bytes) 000433[1] 1721406181.482631 read 0x7692fd thru 0x76acf6 (0x19fa bytes) 000434[1] 1721406181.488174 write 0x8207fe thru 0x829aa5 (0x92a8 bytes) 000439[1] 1721406181.498951 read 0x688a64 thru 0x6922d9 (0x9876 bytes) 000441[0] 1721406181.510326 mapread 0x74c976 thru 0x758f99 (0xc624 bytes) 000442[1] 1721406181.529068 write 0x6cd202 thru 0x6cf529 (0x2328 bytes) 000443[1] 1721406181.529634 read 0x220de2 thru 0x22892d (0x7b4c bytes) 000444[1] 1721406181.549094 read 0x80cf5a thru 0x81cbba (0xfc61 bytes) 000445[0] 1721406181.552349 mapread 0x82f7f2 thru 0x830ef4 (0x1703 bytes) 000446[1] 1721406181.563369 read 0x72d845 thru 0x73987e (0xc03a bytes) 000449[1] 1721406181.564998 mapwrite 0x6ea5a9 thru 0x6eebe9 (0x4641 bytes) 000452[0] 1721406181.591092 write 0x095037 thru 0x09e283 (0x924d bytes) 000455[0] 1721406181.604469 trunc from 0x86537f to 0x7ec35b (0x79023 bytes) 000456[0] 1721406181.613552 mapread 0x4745b2 thru 0x47df85 (0x99d4 bytes) 000457[0] 1721406181.624405 mapwrite 0x3e6d15 thru 0x3f1702 (0xa9ee bytes) 000459[0] 1721406181.651165 read 0x2765a4 thru 0x27d26d (0x6cca bytes) 000460[0] 1721406181.667619 mapread 0x4947d3 thru 0x4a1ca0 (0xd4ce bytes) 000461[1] 1721406181.672160 mapread 0x46c916 thru 0x46ce73 (0x55e bytes) 000463[0] 1721406181.681733 write 0x67f030 thru 0x68b9e7 (0xc9b8 bytes) 000464[0] 1721406181.689257 read 0x29b46f thru 0x2a5056 (0x9be8 bytes) 000465[1] 1721406181.698434 mapwrite 0x08bff0 thru 0x098145 (0xc156 bytes) 000467[1] 1721406181.739551 mapwrite 0x9fc476 thru 0x9fffff (0x3b8a bytes) 000469[0] 1721406181.769967 write 0x5b2da4 thru 0x5ba6bb (0x7918 bytes) 000472[1] 1721406181.784194 read 0x6da8af thru 0x6decc0 (0x4412 bytes) 000473[1] 1721406181.789831 trunc from 0xa00000 to 0x03af82 (0x9c507d bytes) 000474[0] 1721406181.801551 write 0x0151b2 thru 0x022ae4 (0xd933 bytes) 000478[0] 1721406181.818893 mapread 0x024225 thru 0x025938 (0x1714 bytes) 000479[1] 1721406181.825538 write 0x15a19b thru 0x15b222 (0x1088 bytes) 000480[1] 1721406181.829057 trunc from 0x15b223 to 0x5e67b0 (0x48b58e bytes) 000484[0] 1721406181.849391 mapread 0x128534 thru 0x12b417 (0x2ee4 bytes) 000485[1] 1721406181.860100 read 0x5a08b4 thru 0x5a0c62 (0x3af bytes) 000486[0] 1721406181.863046 mapread 0x1c1c79 thru 0x1c7e89 (0x6211 bytes) 000488[1] 1721406181.874699 mapwrite 0x064fbd thru 0x072bb2 (0xdbf6 bytes) 000489[0] 1721406181.909206 trunc from 0x5e67b1 to 0x2a754c (0x33f264 bytes) 000490[1] 1721406181.920123 trunc from 0x2a754d to 0x0a909c (0x1fe4b0 bytes) 000491[0] 1721406181.925819 mapwrite 0x4cd6d6 thru 0x4d092f (0x325a bytes) 000496[0] 1721406181.960522 trunc from 0x4d0930 to 0x228218 (0x2a8717 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 6181 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[0] 1721406182.156753 write_OD 0x19b000 thru 0x1a8fff (0xe000 bytes) truncating to largest ever: 0x86d54a 000003[1] 1721406182.173666 trunc from 0x1a9000 to 0x86d549 (0x6c454a bytes) 000004[1] 1721406182.180252 mapwrite 0x25b000 thru 0x25e20d (0x320e bytes) 000006[1] 1721406182.204385 write_OD 0x722000 thru 0x72cfff (0xb000 bytes) 000007[0] 1721406182.218506 trunc from 0x86d54a to 0x524900 (0x348c49 bytes) 000009[0] 1721406182.230870 trunc from 0x524901 to 0x742e70 (0x21e570 bytes) 000010[1] 1721406182.235321 trunc from 0x742e71 to 0x32f5a0 (0x4138d0 bytes) 000012[0] 1721406182.247995 mapwrite 0x803000 thru 0x80366d (0x66e bytes) 000015[1] 1721406182.272738 write_OD 0x6e9000 thru 0x6effff (0x7000 bytes) 000021[1] 1721406182.288023 mapread 0x3c2000 thru 0x3d1a78 (0xfa79 bytes) 000027[0] 1721406182.294463 mapread 0x0e2000 thru 0x0ecf8b (0xaf8c bytes) 000029[0] 1721406182.300837 mapwrite 0x3b9000 thru 0x3c5a52 (0xca53 bytes) truncating to largest ever: 0x93c5d7 000032[0] 1721406182.330088 trunc from 0x80366e to 0x93c5d6 (0x138f69 bytes) 000033[1] 1721406182.334598 trunc from 0x93c5d7 to 0x7c34f3 (0x1790e3 bytes) 000034[1] 1721406182.343130 mapwrite 0x4d3000 thru 0x4e081d (0xd81e bytes) 000035[1] 1721406182.365357 trunc from 0x7c34f4 to 0x3fc63f (0x3c6eb4 bytes) 000037[1] 1721406182.373904 mapread 0x3de000 thru 0x3e4bf2 (0x6bf3 bytes) 000038[0] 1721406182.379301 trunc from 0x3fc640 to 0x338461 (0xc41de bytes) 000039[0] 1721406182.387273 read_OD 0x1d8000 thru 0x1e4fff (0xd000 bytes) 000040[1] 1721406182.391671 write_OD 0x0db000 thru 0x0e5fff (0xb000 bytes) 000041[1] 1721406182.405242 write_OD 0x183000 thru 0x18bfff (0x9000 bytes) 000042[1] 1721406182.419848 mapread 0x25e000 thru 0x26c265 (0xe266 bytes) 000043[0] 1721406182.426155 mapread 0x2ba000 thru 0x2c55aa (0xb5ab bytes) 000044[1] 1721406182.432542 write_OD 0x4f1000 thru 0x4f6fff (0x6000 bytes) 000045[1] 1721406182.446696 write_OD 0x42a000 thru 0x42efff (0x5000 bytes) 000048[0] 1721406182.461251 trunc from 0x4f7000 to 0x2b1ecb (0x245134 bytes) 000051[0] 1721406182.476306 mapread 0x106000 thru 0x1108a5 (0xa8a6 bytes) 000055[0] 1721406182.493594 read_OD 0x0cf000 thru 0x0ddfff (0xf000 bytes) 000056[1] 1721406182.497780 read_OD 0x2a3000 thru 0x2b0fff (0xe000 bytes) 000059[1] 1721406182.500827 mapread 0x0ba000 thru 0x0c1bf1 (0x7bf2 bytes) 000060[1] 1721406182.505067 mapwrite 0x9aa000 thru 0x9b9945 (0xf946 bytes) 000061[1] 1721406182.532790 read_OD 0x3fc000 thru 0x3fffff (0x4000 bytes) 000063[0] 1721406182.536649 write_OD 0x24b000 thru 0x257fff (0xd000 bytes) 000065[0] 1721406182.554362 read_OD 0x289000 thru 0x297fff (0xf000 bytes) 000068[0] 1721406182.556571 mapread 0x319000 thru 0x31cd7f (0x3d80 bytes) 000074[0] 1721406182.580798 trunc from 0x9b9946 to 0x3048bc (0x6b5089 bytes) 000075[1] 1721406182.596114 mapread 0x043000 thru 0x052d60 (0xfd61 bytes) 000076[0] 1721406182.615062 write_OD 0x668000 thru 0x66efff (0x7000 bytes) 000077[1] 1721406182.630273 mapread 0x310000 thru 0x3120fd (0x20fe bytes) 000081[0] 1721406182.647889 trunc from 0x66f000 to 0x811102 (0x1a2103 bytes) 000082[0] 1721406182.655986 write_OD 0x05a000 thru 0x05cfff (0x3000 bytes) 000083[0] 1721406182.669815 trunc from 0x811103 to 0x11f2a7 (0x6f1e5b bytes) 000084[1] 1721406182.677989 trunc from 0x11f2a8 to 0x72bfce (0x60cd27 bytes) 000085[1] 1721406182.683918 mapread 0x19e000 thru 0x1a2313 (0x4314 bytes) 000086[1] 1721406182.693083 trunc from 0x72bfcf to 0x55bf2f (0x1d009f bytes) 000088[1] 1721406182.701090 mapread 0x007000 thru 0x01114f (0xa150 bytes) 000091[0] 1721406182.719035 mapwrite 0x428000 thru 0x42c821 (0x4822 bytes) 000092[1] 1721406182.744269 mapwrite 0x4dd000 thru 0x4e4794 (0x7795 bytes) 000094[1] 1721406182.773421 write_OD 0x39f000 thru 0x3aafff (0xc000 bytes) 000099[1] 1721406182.789513 mapwrite 0x951000 thru 0x95a1a5 (0x91a6 bytes) 000103[1] 1721406182.823521 read_OD 0x601000 thru 0x60efff (0xe000 bytes) 000105[1] 1721406182.830564 mapwrite 0x490000 thru 0x496381 (0x6382 bytes) 000107[0] 1721406182.865175 trunc from 0x95a1a6 to 0x73ab89 (0x21f61c bytes) 000109[0] 1721406182.877780 trunc from 0x73ab8a to 0x8d8f41 (0x19e3b8 bytes) 000110[0] 1721406182.884822 trunc from 0x8d8f42 to 0x4415cd (0x497974 bytes) 000112[1] 1721406182.893312 mapread 0x0c0000 thru 0x0c4279 (0x427a bytes) 000113[1] 1721406182.901684 mapwrite 0x8dd000 thru 0x8e00c9 (0x30ca bytes) 000117[1] 1721406182.933232 read_OD 0x35d000 thru 0x369fff (0xd000 bytes) 000118[0] 1721406182.937127 mapread 0x0bf000 thru 0x0cba82 (0xca83 bytes) 000121[0] 1721406182.944216 mapread 0x35f000 thru 0x36ad74 (0xbd75 bytes) 000127[0] 1721406182.951020 read_OD 0x6ee000 thru 0x6eefff (0x1000 bytes) 000128[1] 1721406182.952271 write_OD 0x428000 thru 0x428fff (0x1000 bytes) 000129[1] 1721406182.965520 mapwrite 0x6df000 thru 0x6e2d16 (0x3d17 bytes) 000130[1] 1721406183.006244 read_OD 0x74f000 thru 0x758fff (0xa000 bytes) 000131[1] 1721406183.014712 write_OD 0x640000 thru 0x640fff (0x1000 bytes) 000133[0] 1721406183.030739 trunc from 0x8e00ca to 0x8778ee (0x687db bytes) 000135[0] 1721406183.043587 mapread 0x19d000 thru 0x1aa586 (0xd587 bytes) 000136[1] 1721406183.053605 write_OD 0x7c3000 thru 0x7d1fff (0xf000 bytes) 000137[1] 1721406183.068889 mapread 0x1d2000 thru 0x1d4199 (0x219a bytes) 000139[0] 1721406183.078908 trunc from 0x8778ef to 0x3f971e (0x47e1d0 bytes) 000141[1] 1721406183.092219 read_OD 0x271000 thru 0x272fff (0x2000 bytes) 000142[1] 1721406183.097500 write_OD 0x444000 thru 0x449fff (0x6000 bytes) 000143[1] 1721406183.111966 trunc from 0x44a000 to 0x8eb759 (0x4a175a bytes) 000146[1] 1721406183.123278 mapread 0x78f000 thru 0x79138b (0x238c bytes) 000147[0] 1721406183.141205 read_OD 0x031000 thru 0x03efff (0xe000 bytes) 000148[1] 1721406183.152318 write_OD 0x89c000 thru 0x8a9fff (0xe000 bytes) 000149[1] 1721406183.166863 mapwrite 0x5dc000 thru 0x5e3c73 (0x7c74 bytes) 000153[1] 1721406183.205304 mapwrite 0x236000 thru 0x23f751 (0x9752 bytes) 000155[0] 1721406183.237696 mapwrite 0x30b000 thru 0x30c7c6 (0x17c7 bytes) 000156[1] 1721406183.282213 mapwrite 0x75c000 thru 0x76bd50 (0xfd51 bytes) 000157[0] 1721406183.324618 read_OD 0x2e9000 thru 0x2f4fff (0xc000 bytes) 000158[1] 1721406183.330340 mapwrite 0x119000 thru 0x11f894 (0x6895 bytes) 000159[0] 1721406183.352858 mapwrite 0x3ee000 thru 0x3f9d57 (0xbd58 bytes) 000160[0] 1721406183.391098 mapread 0x086000 thru 0x08de44 (0x7e45 bytes) 000161[0] 1721406183.400038 mapread 0x454000 thru 0x463c67 (0xfc68 bytes) 000163[0] 1721406183.403566 mapwrite 0x848000 thru 0x852511 (0xa512 bytes) 000165[0] 1721406183.420965 trunc from 0x8eb75a to 0x53a066 (0x3b16f3 bytes) 000166[1] 1721406183.430488 write_OD 0x39f000 thru 0x3a5fff (0x7000 bytes) 000167[1] 1721406183.446971 mapread 0x08c000 thru 0x096050 (0xa051 bytes) 000169[1] 1721406183.460023 mapwrite 0x5de000 thru 0x5eb040 (0xd041 bytes) 000170[0] 1721406183.496351 mapread 0x4bf000 thru 0x4c3948 (0x4949 bytes) 000173[0] 1721406183.507175 mapwrite 0x1d3000 thru 0x1d9338 (0x6339 bytes) 000174[1] 1721406183.529791 trunc from 0x5eb041 to 0x2e1ccc (0x309374 bytes) 000176[1] 1721406183.540029 write_OD 0x0d0000 thru 0x0d5fff (0x6000 bytes) 000177[1] 1721406183.554209 mapread 0x19f000 thru 0x1a3de5 (0x4de6 bytes) 000179[0] 1721406183.560192 read_OD 0x29d000 thru 0x2a6fff (0xa000 bytes) 000180[0] 1721406183.561941 mapread 0x06a000 thru 0x06ec96 (0x4c97 bytes) 000183[1] 1721406183.568801 write_OD 0x08c000 thru 0x08cfff (0x1000 bytes) 000184[0] 1721406183.581687 mapwrite 0x70b000 thru 0x71142b (0x642c bytes) 000185[1] 1721406183.616794 trunc from 0x71142c to 0x019f86 (0x6f74a5 bytes) 000187[0] 1721406183.629117 write_OD 0x3e0000 thru 0x3ecfff (0xd000 bytes) 000188[0] 1721406183.643066 mapread 0x0ce000 thru 0x0dd65d (0xf65e bytes) 000190[1] 1721406183.661443 mapwrite 0x2a8000 thru 0x2a9482 (0x1483 bytes) 000191[1] 1721406183.679307 trunc from 0x3ed000 to 0x11fe5f (0x2cd1a0 bytes) 000192[0] 1721406183.683886 mapread 0x09c000 thru 0x0a6d00 (0xad01 bytes) 000193[1] 1721406183.700298 trunc from 0x11fe60 to 0x29bad4 (0x17bc75 bytes) 000202[1] 1721406183.714686 mapread 0x1a0000 thru 0x1aca60 (0xca61 bytes) 000204[0] 1721406183.734077 mapread 0x167000 thru 0x16a3d5 (0x33d6 bytes) 000206[0] 1721406183.742597 mapwrite 0x7cc000 thru 0x7db8ff (0xf900 bytes) 000207[1] 1721406183.780527 mapread 0x20d000 thru 0x21995a (0xc95b bytes) 000209[1] 1721406183.798949 trunc from 0x7db900 to 0x3e6076 (0x3f5889 bytes) 000214[0] 1721406183.814611 trunc from 0x3e6077 to 0x0799fc (0x36c67a bytes) 000215[0] 1721406183.823396 write_OD 0x1fb000 thru 0x206fff (0xc000 bytes) 000217[1] 1721406183.838258 write_OD 0x491000 thru 0x499fff (0x9000 bytes) 000218[1] 1721406183.852971 write_OD 0x922000 thru 0x92ffff (0xe000 bytes) 000219[1] 1721406183.868355 trunc from 0x930000 to 0x4d7b33 (0x4584cc bytes) 000221[1] 1721406183.877874 trunc from 0x4d7b34 to 0x455926 (0x8220d bytes) 000224[1] 1721406183.891758 trunc from 0x455927 to 0x2ad148 (0x1a87de bytes) 000226[1] 1721406183.896483 mapwrite 0x3c3000 thru 0x3c99c4 (0x69c5 bytes) 000227[1] 1721406183.921261 write_OD 0x53c000 thru 0x53ffff (0x4000 bytes) 000229[1] 1721406183.938959 write_OD 0x91d000 thru 0x91dfff (0x1000 bytes) 000232[0] 1721406183.953137 mapread 0x1fa000 thru 0x2088a8 (0xe8a9 bytes) 000233[1] 1721406183.968293 mapread 0x4c4000 thru 0x4cdacb (0x9acc bytes) 000236[0] 1721406183.980854 write_OD 0x944000 thru 0x94afff (0x7000 bytes) 000237[1] 1721406183.996322 write_OD 0x9b6000 thru 0x9c3fff (0xe000 bytes) truncating to largest ever: 0x97c722 000239[0] 1721406184.012525 trunc from 0x9c4000 to 0x97c721 (0x478de bytes) 000241[0] 1721406184.024479 trunc from 0x97c722 to 0x1e21d1 (0x79a550 bytes) 000242[0] 1721406184.031586 mapread 0x1ba000 thru 0x1be507 (0x4508 bytes) 000244[0] 1721406184.040841 trunc from 0x1e21d2 to 0x027b02 (0x1ba6cf bytes) 000246[0] 1721406184.045115 write_OD 0x302000 thru 0x309fff (0x8000 bytes) 000252[0] 1721406184.066265 mapwrite 0x292000 thru 0x2943b2 (0x23b3 bytes) 000253[0] 1721406184.087649 trunc from 0x30a000 to 0x386a37 (0x7ca38 bytes) 000254[1] 1721406184.091810 trunc from 0x386a38 to 0x20ab84 (0x17beb3 bytes) 000257[1] 1721406184.103973 mapwrite 0x7da000 thru 0x7e7bd1 (0xdbd2 bytes) 000259[0] 1721406184.144503 write_OD 0x46f000 thru 0x470fff (0x2000 bytes) 000260[1] 1721406184.158183 trunc from 0x7e7bd2 to 0x8375ae (0x4f9dd bytes) 000261[0] 1721406184.163735 mapwrite 0x8e2000 thru 0x8e849c (0x649d bytes) 000262[0] 1721406184.191096 trunc from 0x8e849d to 0x675da5 (0x2726f7 bytes) 000264[0] 1721406184.194405 write_OD 0x570000 thru 0x573fff (0x4000 bytes) 000266[0] 1721406184.211981 read_OD 0x187000 thru 0x194fff (0xe000 bytes) 000267[1] 1721406184.214087 mapwrite 0x50d000 thru 0x518729 (0xb72a bytes) 000270[0] 1721406184.252556 mapread 0x1c8000 thru 0x1d3350 (0xb351 bytes) 000271[0] 1721406184.270740 mapwrite 0x2c8000 thru 0x2d19a0 (0x99a1 bytes) 000273[0] 1721406184.313454 mapwrite 0x7e0000 thru 0x7ec86a (0xc86b bytes) 000275[0] 1721406184.346646 trunc from 0x7ec86b to 0x6e26f8 (0x10a172 bytes) 000277[1] 1721406184.353746 trunc from 0x6e26f9 to 0x37527b (0x36d47d bytes) 000278[0] 1721406184.359669 write_OD 0x2a9000 thru 0x2b6fff (0xe000 bytes) 000279[0] 1721406184.373689 mapwrite 0x27c000 thru 0x28128d (0x528e bytes) truncating to largest ever: 0x9f9a0f 000282[1] 1721406184.400789 trunc from 0x37527c to 0x9f9a0e (0x684793 bytes) 000285[1] 1721406184.414147 mapread 0x254000 thru 0x258680 (0x4681 bytes) 000288[1] 1721406184.427188 mapread 0x841000 thru 0x845728 (0x4729 bytes) 000289[1] 1721406184.432219 mapwrite 0x7c6000 thru 0x7c921a (0x321b bytes) 000290[0] 1721406184.449985 read_OD 0x4bb000 thru 0x4c2fff (0x8000 bytes) 000291[0] 1721406184.456027 mapwrite 0x4b0000 thru 0x4be55f (0xe560 bytes) 000294[1] 1721406184.505757 mapread 0x372000 thru 0x372d14 (0xd15 bytes) 000297[0] 1721406184.515865 write_OD 0x513000 thru 0x521fff (0xf000 bytes) 000303[0] 1721406184.532026 trunc from 0x9f9a0f to 0x0581c3 (0x9a184b bytes) 000304[1] 1721406184.541305 write_OD 0x2e7000 thru 0x2e8fff (0x2000 bytes) 000305[1] 1721406184.555022 read_OD 0x097000 thru 0x098fff (0x2000 bytes) 000307[0] 1721406184.558177 write_OD 0x987000 thru 0x994fff (0xe000 bytes) 000309[1] 1721406184.575163 mapread 0x255000 thru 0x261c48 (0xcc49 bytes) 000310[0] 1721406184.599668 mapwrite 0x345000 thru 0x34fdfc (0xadfd bytes) 000311[0] 1721406184.637122 read_OD 0x0cd000 thru 0x0cefff (0x2000 bytes) 000312[1] 1721406184.640796 read_OD 0x538000 thru 0x53afff (0x3000 bytes) 000313[1] 1721406184.642861 read_OD 0x518000 thru 0x519fff (0x2000 bytes) 000314[1] 1721406184.644254 trunc from 0x995000 to 0x528a6a (0x46c595 bytes) 000316[1] 1721406184.650312 write_OD 0x6b0000 thru 0x6b0fff (0x1000 bytes) 000317[0] 1721406184.665014 trunc from 0x6b1000 to 0x7fc51b (0x14b51c bytes) 000318[0] 1721406184.673823 trunc from 0x7fc51c to 0x83856e (0x3c053 bytes) 000320[0] 1721406184.682126 mapread 0x57d000 thru 0x58981f (0xc820 bytes) 000321[0] 1721406184.703399 write_OD 0x86c000 thru 0x86cfff (0x1000 bytes) 000322[0] 1721406184.723347 mapwrite 0x97e000 thru 0x982508 (0x4509 bytes) 000323[1] 1721406184.755042 read_OD 0x5c1000 thru 0x5c5fff (0x5000 bytes) 000324[1] 1721406184.760940 read_OD 0x0f2000 thru 0x0f2fff (0x1000 bytes) 000326[0] 1721406184.764350 write_OD 0x13a000 thru 0x13ffff (0x6000 bytes) 000328[0] 1721406184.781757 read_OD 0x744000 thru 0x74dfff (0xa000 bytes) 000330[1] 1721406184.784731 mapwrite 0x2cc000 thru 0x2d0170 (0x4171 bytes) 000332[1] 1721406184.832714 trunc from 0x982509 to 0x043659 (0x93eeaf bytes) 000333[0] 1721406184.847794 mapread 0x014000 thru 0x01ee06 (0xae07 bytes) 000336[0] 1721406184.864946 mapwrite 0x145000 thru 0x1533ff (0xe400 bytes) 000337[1] 1721406184.904679 write_OD 0x44f000 thru 0x457fff (0x9000 bytes) 000339[0] 1721406184.922531 mapwrite 0x124000 thru 0x1264bd (0x24be bytes) 000340[1] 1721406184.944340 mapwrite 0x02d000 thru 0x02eeb0 (0x1eb1 bytes) 000343[0] 1721406184.969936 trunc from 0x458000 to 0x2c3b8d (0x194472 bytes) 000345[0] 1721406184.982372 read_OD 0x26b000 thru 0x26cfff (0x2000 bytes) 000346[1] 1721406184.985813 write_OD 0x606000 thru 0x60afff (0x5000 bytes) 000347[1] 1721406185.004267 mapwrite 0x5c3000 thru 0x5cce63 (0x9e64 bytes) 000349[0] 1721406185.036623 mapread 0x205000 thru 0x21444a (0xf44b bytes) 000350[1] 1721406185.067010 trunc from 0x60b000 to 0x4e4a88 (0x126577 bytes) 000352[1] 1721406185.084032 mapwrite 0x766000 thru 0x76bcfa (0x5cfb bytes) 000353[1] 1721406185.111678 write_OD 0x361000 thru 0x367fff (0x7000 bytes) 000355[0] 1721406185.128073 mapwrite 0x6bd000 thru 0x6c9180 (0xc181 bytes) 000356[1] 1721406185.154503 write_OD 0x935000 thru 0x938fff (0x4000 bytes) 000357[1] 1721406185.168174 trunc from 0x939000 to 0x78ae85 (0x1ae17a bytes) 000360[0] 1721406185.177781 mapread 0x4c9000 thru 0x4d2cb9 (0x9cba bytes) 000361[1] 1721406185.192012 mapread 0x6c0000 thru 0x6cb461 (0xb462 bytes) 000362[1] 1721406185.204071 write_OD 0x3cb000 thru 0x3d6fff (0xc000 bytes) 000368[1] 1721406185.220566 mapwrite 0x55c000 thru 0x568fc2 (0xcfc3 bytes) 000369[0] 1721406185.253277 read_OD 0x031000 thru 0x03cfff (0xc000 bytes) 000370[0] 1721406185.258547 mapread 0x10c000 thru 0x11b460 (0xf461 bytes) 000371[1] 1721406185.280119 write_OD 0x660000 thru 0x66afff (0xb000 bytes) 000372[1] 1721406185.298606 write_OD 0x16a000 thru 0x175fff (0xc000 bytes) 000374[1] 1721406185.312905 read_OD 0x57d000 thru 0x584fff (0x8000 bytes) 000375[0] 1721406185.314896 read_OD 0x36b000 thru 0x36bfff (0x1000 bytes) 000381[1] 1721406185.316540 write_OD 0x392000 thru 0x39dfff (0xc000 bytes) 000382[1] 1721406185.330030 mapwrite 0x0af000 thru 0x0afa02 (0xa03 bytes) 000383[1] 1721406185.357143 read_OD 0x4bb000 thru 0x4bcfff (0x2000 bytes) 000387[0] 1721406185.364091 read_OD 0x498000 thru 0x4a0fff (0x9000 bytes) 000388[1] 1721406185.366176 mapwrite 0x4ae000 thru 0x4b9231 (0xb232 bytes) 000390[1] 1721406185.404742 trunc from 0x78ae86 to 0x7dda01 (0x52b7c bytes) 000392[1] 1721406185.415368 trunc from 0x7dda02 to 0x8fe02b (0x12062a bytes) 000394[0] 1721406185.425242 trunc from 0x8fe02c to 0x92432b (0x26300 bytes) 000395[0] 1721406185.434948 mapwrite 0x0fe000 thru 0x10749c (0x949d bytes) 000396[0] 1721406185.475840 mapwrite 0x277000 thru 0x27aa5a (0x3a5b bytes) 000397[1] 1721406185.494885 read_OD 0x4f1000 thru 0x4fcfff (0xc000 bytes) 000398[0] 1721406185.504281 write_OD 0x5af000 thru 0x5b7fff (0x9000 bytes) 000401[1] 1721406185.517694 mapwrite 0x0f1000 thru 0x0f991b (0x891c bytes) 000402[1] 1721406185.553013 trunc from 0x92432c to 0x602456 (0x321ed5 bytes) 000403[1] 1721406185.559126 trunc from 0x602457 to 0x559a9a (0xa89bc bytes) 000404[0] 1721406185.562529 mapwrite 0x202000 thru 0x20a557 (0x8558 bytes) 000405[0] 1721406185.589939 mapread 0x524000 thru 0x5240ff (0x100 bytes) 000408[0] 1721406185.594398 trunc from 0x559a9b to 0x58afce (0x31534 bytes) 000409[1] 1721406185.598569 read_OD 0x52a000 thru 0x533fff (0xa000 bytes) 000410[1] 1721406185.604708 mapread 0x3dd000 thru 0x3e73b3 (0xa3b4 bytes) 000417[0] 1721406185.619067 read_OD 0x21b000 thru 0x220fff (0x6000 bytes) 000419[0] 1721406185.621512 mapwrite 0x9ac000 thru 0x9b91c8 (0xd1c9 bytes) 000423[1] 1721406185.670651 mapwrite 0x8c5000 thru 0x8c7146 (0x2147 bytes) 000424[1] 1721406185.709813 trunc from 0x9b91c9 to 0x99182c (0x2799c bytes) 000427[0] 1721406185.721646 read_OD 0x4d4000 thru 0x4dbfff (0x8000 bytes) 000428[0] 1721406185.730000 trunc from 0x99182d to 0x951eb0 (0x3f97c bytes) 000429[0] 1721406185.735523 trunc from 0x951eb1 to 0x402b9c (0x54f314 bytes) 000430[1] 1721406185.741811 mapwrite 0x5d0000 thru 0x5df5cc (0xf5cd bytes) 000431[1] 1721406185.778109 write_OD 0x8f2000 thru 0x8fcfff (0xb000 bytes) 000433[1] 1721406185.794304 read_OD 0x4f2000 thru 0x4f4fff (0x3000 bytes) 000435[1] 1721406185.795889 mapwrite 0x5fe000 thru 0x607c3f (0x9c40 bytes) 000437[1] 1721406185.834048 read_OD 0x6c3000 thru 0x6c3fff (0x1000 bytes) 000441[1] 1721406185.842212 mapread 0x127000 thru 0x135620 (0xe621 bytes) 000442[1] 1721406185.867017 trunc from 0x8fd000 to 0x276444 (0x686bbb bytes) 000446[1] 1721406185.882489 write_OD 0x195000 thru 0x19efff (0xa000 bytes) 000447[1] 1721406185.897135 write_OD 0x626000 thru 0x62ffff (0xa000 bytes) 000450[0] 1721406185.912161 read_OD 0x36d000 thru 0x36efff (0x2000 bytes) 000452[1] 1721406185.913539 mapwrite 0x00b000 thru 0x013407 (0x8408 bytes) 000453[0] 1721406185.938324 mapread 0x0e3000 thru 0x0eab53 (0x7b54 bytes) 000455[1] 1721406185.960085 write_OD 0x492000 thru 0x492fff (0x1000 bytes) 000457[0] 1721406185.978182 trunc from 0x630000 to 0x4c0e96 (0x16f169 bytes) 000458[1] 1721406185.984504 write_OD 0x840000 thru 0x848fff (0x9000 bytes) 000459[1] 1721406185.999694 read_OD 0x063000 thru 0x064fff (0x2000 bytes) 000460[1] 1721406186.002834 write_OD 0x08b000 thru 0x090fff (0x6000 bytes) 000461[0] 1721406186.016687 read_OD 0x121000 thru 0x124fff (0x4000 bytes) 000463[0] 1721406186.019036 write_OD 0x44c000 thru 0x452fff (0x7000 bytes) 000464[0] 1721406186.034051 write_OD 0x5f7000 thru 0x602fff (0xc000 bytes) 000469[1] 1721406186.050318 read_OD 0x5ae000 thru 0x5bcfff (0xf000 bytes) 000471[1] 1721406186.052634 read_OD 0x523000 thru 0x52bfff (0x9000 bytes) 000472[0] 1721406186.054539 trunc from 0x849000 to 0x4f8cde (0x350321 bytes) 000473[0] 1721406186.060677 write_OD 0x71b000 thru 0x71bfff (0x1000 bytes) 000475[1] 1721406186.078957 write_OD 0x8a1000 thru 0x8a1fff (0x1000 bytes) 000477[1] 1721406186.093926 write_OD 0x89f000 thru 0x8a2fff (0x4000 bytes) 000479[0] 1721406186.111423 mapwrite 0x945000 thru 0x94bfe3 (0x6fe4 bytes) 000480[0] 1721406186.159027 write_OD 0x610000 thru 0x613fff (0x4000 bytes) 000483[1] 1721406186.181168 read_OD 0x8f8000 thru 0x8fdfff (0x6000 bytes) 000484[0] 1721406186.183024 read_OD 0x5f4000 thru 0x601fff (0xe000 bytes) 000485[0] 1721406186.184868 read_OD 0x695000 thru 0x696fff (0x2000 bytes) 000486[0] 1721406186.186184 read_OD 0x523000 thru 0x524fff (0x2000 bytes) 000487[1] 1721406186.187452 write_OD 0x64f000 thru 0x65cfff (0xe000 bytes) 000488[0] 1721406186.201275 read_OD 0x73a000 thru 0x746fff (0xd000 bytes) 000490[0] 1721406186.204111 trunc from 0x94bfe4 to 0x283106 (0x6c8edd bytes) 000492[0] 1721406186.216308 trunc from 0x283107 to 0x76f22b (0x4ec125 bytes) 000493[0] 1721406186.224119 write_OD 0x7ac000 thru 0x7affff (0x4000 bytes) 000494[1] 1721406186.239900 mapwrite 0x5e7000 thru 0x5f600e (0xf00f bytes) 000499[1] 1721406186.285938 trunc from 0x7b0000 to 0x892e6e (0xe2e6f bytes) All operations completed A-OK! PASS 16a (11s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 12:23:07 (1721406187) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0913109 s, 23.0 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0578367 s, 36.3 MB/s Chance of close/open is 1 in 50 Seed set to 6187 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3ad4b4 000001[0] 1721406188.098235 trunc from 00000000 to 0x3ad4b3 (0x3ad4b4 bytes) 000002[0] 1721406188.105818 write 0x93e437 thru 0x93e8bb (0x485 bytes) truncating to largest ever: 0x50e235 000004[1] 1721406188.111507 trunc from 0x93e8bc to 0x50e234 (0x430687 bytes) truncating to largest ever: 0x84455a 000006[0] 1721406188.124631 trunc from 0x50e235 to 0x844559 (0x336325 bytes) 000007[1] 1721406188.134028 write 0x932c66 thru 0x933dd8 (0x1173 bytes) 000008[0] 1721406188.140053 trunc from 0x933dd9 to 0x3fd637 (0x5367a1 bytes) 000010[0] 1721406188.155839 read 0x23f991 thru 0x23fd3d (0x3ad bytes) 000012[0] 1721406188.161604 mapwrite 0x3e69cd thru 0x3e8226 (0x185a bytes) 000015[1] 1721406188.192883 mapread 0x2daeee thru 0x2dc698 (0x17ab bytes) 000020[1] 1721406188.212057 read 0x2e9956 thru 0x2e9a3f (0xea bytes) 000021[1] 1721406188.216454 mapwrite 0x07eab9 thru 0x080797 (0x1cdf bytes) 000022[0] 1721406188.243565 mapwrite 0x1ca107 thru 0x1caefb (0xdf5 bytes) 000023[0] 1721406188.265188 mapread 0x25490e thru 0x254c35 (0x328 bytes) truncating to largest ever: 0x9439a7 000028[0] 1721406188.277959 trunc from 0x3fd638 to 0x9439a6 (0x54636f bytes) 000031[1] 1721406188.294337 write 0x578312 thru 0x578d9d (0xa8c bytes) 000033[1] 1721406188.305291 trunc from 0x9439a7 to 0x864fbf (0xde9e7 bytes) 000034[0] 1721406188.312321 mapwrite 0x3a5284 thru 0x3a6188 (0xf05 bytes) 000039[1] 1721406188.352103 read 0x54adf1 thru 0x54b622 (0x832 bytes) 000040[1] 1721406188.373283 write 0x9c1105 thru 0x9c1781 (0x67d bytes) 000042[0] 1721406188.377863 mapread 0x30872d thru 0x309173 (0xa47 bytes) 000043[1] 1721406188.387987 mapread 0x453633 thru 0x45386b (0x239 bytes) 000044[1] 1721406188.394562 mapread 0x4ab37f thru 0x4aca1e (0x16a0 bytes) 000048[1] 1721406188.402317 write 0x3701ff thru 0x371005 (0xe07 bytes) 000049[0] 1721406188.407425 trunc from 0x9c1782 to 0x79b72b (0x226056 bytes) 000050[0] 1721406188.419184 mapwrite 0x656b47 thru 0x658887 (0x1d41 bytes) 000051[0] 1721406188.443002 mapread 0x2ddec7 thru 0x2de714 (0x84e bytes) 000053[0] 1721406188.450699 mapread 0x404b3a thru 0x406210 (0x16d7 bytes) 000054[0] 1721406188.458781 trunc from 0x79b72c to 0x4df63d (0x2bc0ee bytes) truncating to largest ever: 0x9d249c 000055[0] 1721406188.467257 trunc from 0x4df63e to 0x9d249b (0x4f2e5e bytes) 000056[0] 1721406188.472100 write 0x7d09ae thru 0x7d1827 (0xe7a bytes) 000057[1] 1721406188.474364 trunc from 0x9d249c to 0x2f0658 (0x6e1e43 bytes) 000058[1] 1721406188.482786 mapwrite 0x988322 thru 0x988377 (0x56 bytes) 000059[1] 1721406188.504715 trunc from 0x988378 to 0x81f0ae (0x1692c9 bytes) 000060[1] 1721406188.512907 read 0x15c8ff thru 0x15df0f (0x1611 bytes) 000061[0] 1721406188.546652 read 0x3041db thru 0x305ca8 (0x1ace bytes) 000062[1] 1721406188.567701 mapread 0x699e03 thru 0x69b44e (0x164c bytes) 000063[1] 1721406188.568997 read 0x56307e thru 0x5647be (0x1741 bytes) 000064[1] 1721406188.583227 read 0x0f8035 thru 0x0f9e83 (0x1e4f bytes) 000066[1] 1721406188.600709 mapwrite 0x78015b thru 0x780543 (0x3e9 bytes) 000068[1] 1721406188.639125 mapwrite 0x3f9ae9 thru 0x3fa2ca (0x7e2 bytes) 000070[1] 1721406188.665602 write 0x456916 thru 0x4586f2 (0x1ddd bytes) 000075[0] 1721406188.673693 mapread 0x1dbc36 thru 0x1dccca (0x1095 bytes) 000076[0] 1721406188.684118 mapwrite 0x24640e thru 0x246f47 (0xb3a bytes) 000077[0] 1721406188.728094 mapwrite 0x32d894 thru 0x32e1ca (0x937 bytes) 000079[1] 1721406188.752802 mapread 0x3cbc91 thru 0x3cd094 (0x1404 bytes) 000080[0] 1721406188.764233 mapread 0x204907 thru 0x205c6c (0x1366 bytes) 000081[1] 1721406188.771023 write 0x9a7448 thru 0x9a7677 (0x230 bytes) 000084[0] 1721406188.784374 read 0x8986a6 thru 0x899c43 (0x159e bytes) 000086[0] 1721406188.797490 write 0x84737e thru 0x848a12 (0x1695 bytes) 000092[0] 1721406188.817368 mapwrite 0x41ff1d thru 0x420b8d (0xc71 bytes) 000093[1] 1721406188.839615 trunc from 0x9a7678 to 0x2cd2eb (0x6da38c bytes) 000095[0] 1721406188.852956 trunc from 0x2cd2ec to 0x544e2c (0x277b41 bytes) 000096[0] 1721406188.861922 write 0x1f4c3c thru 0x1f5028 (0x3ed bytes) 000097[1] 1721406188.868454 write 0x665a9d thru 0x666cca (0x122e bytes) 000104[0] 1721406188.894357 mapwrite 0x887f80 thru 0x888ee2 (0xf63 bytes) 000105[0] 1721406188.919087 trunc from 0x888ee3 to 0x2e53f0 (0x5a3af2 bytes) 000108[0] 1721406188.922894 trunc from 0x2e53f1 to 0x6d2053 (0x3ecc63 bytes) 000109[0] 1721406188.928457 write 0x9e3a81 thru 0x9e4d25 (0x12a5 bytes) 000110[1] 1721406188.935134 mapwrite 0x5f034d thru 0x5f126b (0xf1f bytes) 000111[0] 1721406188.961695 mapwrite 0x037fa8 thru 0x038418 (0x471 bytes) 000113[1] 1721406188.985686 mapread 0x55d714 thru 0x55db5a (0x447 bytes) 000115[0] 1721406188.994687 mapread 0x00eda9 thru 0x00fcfd (0xf55 bytes) 000117[1] 1721406189.005477 read 0x295f3f thru 0x297088 (0x114a bytes) 000121[1] 1721406189.041829 mapwrite 0x914428 thru 0x915f61 (0x1b3a bytes) 000122[1] 1721406189.060363 read 0x554b9b thru 0x55657e (0x19e4 bytes) 000123[1] 1721406189.060697 write 0x1b6a87 thru 0x1b8278 (0x17f2 bytes) 000124[0] 1721406189.067273 mapwrite 0x77078d thru 0x772593 (0x1e07 bytes) 000125[1] 1721406189.098775 read 0x4df6e9 thru 0x4e0387 (0xc9f bytes) 000129[0] 1721406189.102571 trunc from 0x9e4d26 to 0x97d1a4 (0x67b81 bytes) 000130[0] 1721406189.115309 read 0x7598b5 thru 0x7599e3 (0x12f bytes) 000131[0] 1721406189.130628 trunc from 0x97d1a5 to 0x85c728 (0x120a7c bytes) 000132[1] 1721406189.136590 trunc from 0x85c729 to 0x0c3248 (0x7994e0 bytes) 000133[0] 1721406189.149114 mapread 0x07fdaa thru 0x08158f (0x17e6 bytes) 000134[0] 1721406189.160688 read 0x0668a7 thru 0x0676ac (0xe06 bytes) 000135[0] 1721406189.167454 mapwrite 0x639fef thru 0x63a264 (0x276 bytes) 000137[1] 1721406189.216248 mapwrite 0x913d81 thru 0x91468e (0x90e bytes) 000140[0] 1721406189.259765 trunc from 0x91468f to 0x1ed77f (0x726f0f bytes) 000142[0] 1721406189.273793 mapwrite 0x6a3b5a thru 0x6a534c (0x17f3 bytes) 000146[1] 1721406189.309791 mapwrite 0x042b41 thru 0x042db9 (0x279 bytes) 000150[1] 1721406189.344733 read 0x5e1907 thru 0x5e1964 (0x5e bytes) 000151[0] 1721406189.355374 mapread 0x5601db thru 0x56029e (0xc4 bytes) 000154[0] 1721406189.358752 mapwrite 0x57ba9a thru 0x57bfdc (0x543 bytes) 000156[0] 1721406189.387252 trunc from 0x6a534d to 0x32c199 (0x3791b3 bytes) 000157[0] 1721406189.394331 mapwrite 0x727f4d thru 0x7281d4 (0x288 bytes) 000158[0] 1721406189.419784 mapread 0x536d2f thru 0x537e88 (0x115a bytes) 000159[1] 1721406189.426827 write 0x83f3ea thru 0x840bc5 (0x17dc bytes) 000161[0] 1721406189.437154 trunc from 0x840bc6 to 0x0ec4df (0x7546e6 bytes) 000163[1] 1721406189.449674 read 0x04e8be thru 0x05088c (0x1fcf bytes) 000164[0] 1721406189.458074 trunc from 0x0ec4e0 to 0x734abc (0x6485dd bytes) 000165[1] 1721406189.468647 write 0x596dc9 thru 0x597375 (0x5ad bytes) 000166[1] 1721406189.475818 mapread 0x5636c5 thru 0x564f13 (0x184f bytes) 000167[0] 1721406189.482698 mapwrite 0x88cf1f thru 0x88e2bb (0x139d bytes) 000168[0] 1721406189.511563 mapwrite 0x1094a8 thru 0x10ac07 (0x1760 bytes) 000169[0] 1721406189.536421 mapread 0x4e9b3e thru 0x4ea762 (0xc25 bytes) 000174[0] 1721406189.550484 mapwrite 0x8ab617 thru 0x8ac94a (0x1334 bytes) 000179[1] 1721406189.581168 mapwrite 0x5fe6b1 thru 0x5fea76 (0x3c6 bytes) 000180[1] 1721406189.608896 write 0x980eef thru 0x982799 (0x18ab bytes) 000181[1] 1721406189.613719 mapread 0x564c20 thru 0x5654c7 (0x8a8 bytes) 000182[1] 1721406189.621696 write 0x7ea44a thru 0x7ec2c5 (0x1e7c bytes) 000184[0] 1721406189.631697 read 0x08267c thru 0x08455b (0x1ee0 bytes) 000185[0] 1721406189.640950 write 0x061a8e thru 0x0638a0 (0x1e13 bytes) 000193[0] 1721406189.671381 trunc from 0x98279a to 0x6f59a4 (0x28cdf5 bytes) 000194[0] 1721406189.684860 mapread 0x59b57e thru 0x59cb02 (0x1585 bytes) 000195[0] 1721406189.714346 mapwrite 0x313ffc thru 0x315aff (0x1b04 bytes) 000196[1] 1721406189.732032 read 0x44a2b8 thru 0x44b578 (0x12c1 bytes) 000197[0] 1721406189.748804 write 0x8786a8 thru 0x87a56a (0x1ec3 bytes) 000199[0] 1721406189.755450 read 0x6f82a5 thru 0x6f9fb6 (0x1d12 bytes) 000200[1] 1721406189.763874 write 0x966716 thru 0x967430 (0xd1b bytes) 000202[0] 1721406189.776618 mapwrite 0x790fdf thru 0x791283 (0x2a5 bytes) 000204[1] 1721406189.810079 mapread 0x02155d thru 0x021a51 (0x4f5 bytes) 000205[0] 1721406189.821089 read 0x36d0aa thru 0x36e76d (0x16c4 bytes) 000206[0] 1721406189.831308 read 0x775a5b thru 0x775fb5 (0x55b bytes) 000207[0] 1721406189.841208 mapwrite 0x343a74 thru 0x344c11 (0x119e bytes) 000209[1] 1721406189.863099 write 0x87ed8c thru 0x87f3c0 (0x635 bytes) 000210[1] 1721406189.870106 read 0x54bd5b thru 0x54c36f (0x615 bytes) 000212[0] 1721406189.903174 read 0x42161e thru 0x4231d4 (0x1bb7 bytes) 000214[0] 1721406189.927498 mapwrite 0x5cd755 thru 0x5ced46 (0x15f2 bytes) 000217[0] 1721406189.959111 mapread 0x0fb5ae thru 0x0fc104 (0xb57 bytes) 000218[0] 1721406189.964636 mapwrite 0x23784f thru 0x237afa (0x2ac bytes) 000219[0] 1721406189.986145 mapread 0x572bf2 thru 0x5740c1 (0x14d0 bytes) 000220[1] 1721406189.989767 mapwrite 0x693604 thru 0x693ea5 (0x8a2 bytes) 000221[0] 1721406190.011672 trunc from 0x967431 to 0x34fb04 (0x61792c bytes) 000223[0] 1721406190.022153 mapread 0x3226a6 thru 0x32283f (0x19a bytes) 000224[0] 1721406190.023669 write 0x7f8c79 thru 0x7fa574 (0x18fc bytes) 000227[0] 1721406190.028864 mapread 0x753586 thru 0x75532f (0x1daa bytes) 000230[1] 1721406190.039477 write 0x104e5c thru 0x10521e (0x3c3 bytes) 000231[0] 1721406190.048556 trunc from 0x7fa575 to 0x25e771 (0x59be03 bytes) 000233[0] 1721406190.057058 write 0x1d7ad4 thru 0x1d8934 (0xe61 bytes) 000234[1] 1721406190.059632 mapwrite 0x97cc9c thru 0x97e0df (0x1444 bytes) 000236[0] 1721406190.096492 mapread 0x6cf7b0 thru 0x6d11dc (0x1a2d bytes) 000238[0] 1721406190.107830 trunc from 0x97e0e0 to 0x72fdde (0x24e301 bytes) 000239[1] 1721406190.119198 mapwrite 0x8635e0 thru 0x8654a9 (0x1eca bytes) 000240[1] 1721406190.144824 read 0x04bb8f thru 0x04c7ab (0xc1d bytes) 000241[1] 1721406190.150683 write 0x04ef8f thru 0x050536 (0x15a8 bytes) 000246[1] 1721406190.169276 mapread 0x50c2bf thru 0x50c75e (0x4a0 bytes) 000247[0] 1721406190.170996 mapwrite 0x2cb8b2 thru 0x2cc9b1 (0x1100 bytes) 000251[1] 1721406190.203993 read 0x221ee2 thru 0x222771 (0x890 bytes) 000253[0] 1721406190.230402 trunc from 0x8654aa to 0x33fc4c (0x52585d bytes) 000254[0] 1721406190.248185 write 0x33b009 thru 0x33c797 (0x178f bytes) 000255[1] 1721406190.251310 mapwrite 0x7051ab thru 0x705688 (0x4de bytes) 000256[0] 1721406190.276407 mapread 0x1eddd0 thru 0x1efaad (0x1cde bytes) 000257[0] 1721406190.284088 mapwrite 0x88ea6a thru 0x88fa18 (0xfaf bytes) 000258[1] 1721406190.307388 mapwrite 0x648f29 thru 0x64a394 (0x146c bytes) 000261[1] 1721406190.341745 trunc from 0x88fa19 to 0x7daa0e (0xb500a bytes) 000262[0] 1721406190.351368 trunc from 0x7daa0f to 0x64edd7 (0x18bc37 bytes) 000263[1] 1721406190.358488 write 0x601a15 thru 0x6024d6 (0xac2 bytes) 000264[0] 1721406190.367418 write 0x9de918 thru 0x9dfcda (0x13c3 bytes) 000265[1] 1721406190.372129 trunc from 0x9dfcdb to 0x897200 (0x148ada bytes) 000266[1] 1721406190.383278 read 0x82cdd4 thru 0x82ddce (0xffb bytes) 000267[0] 1721406190.391230 trunc from 0x897201 to 0x2a8459 (0x5eeda7 bytes) 000271[0] 1721406190.411460 write 0x47a265 thru 0x47b947 (0x16e3 bytes) 000272[0] 1721406190.415728 trunc from 0x47b948 to 0x116f6a (0x3649dd bytes) 000273[1] 1721406190.420351 write 0x295d8e thru 0x296191 (0x404 bytes) 000274[1] 1721406190.426276 trunc from 0x296192 to 0x7d8ca6 (0x542b15 bytes) 000276[1] 1721406190.439364 trunc from 0x7d8ca7 to 0x8f866a (0x11f9c4 bytes) 000279[1] 1721406190.452069 read 0x43b86e thru 0x43bd66 (0x4f9 bytes) 000280[0] 1721406190.468134 read 0x017579 thru 0x017f96 (0xa1e bytes) 000282[1] 1721406190.484395 mapread 0x132f31 thru 0x13375e (0x82e bytes) 000283[0] 1721406190.490604 mapwrite 0x091543 thru 0x092f17 (0x19d5 bytes) 000285[0] 1721406190.535521 trunc from 0x8f866b to 0x295181 (0x6634e9 bytes) 000287[1] 1721406190.544758 mapread 0x13d007 thru 0x13eaf5 (0x1aef bytes) 000289[1] 1721406190.557442 mapread 0x28523a thru 0x2868b3 (0x167a bytes) 000293[0] 1721406190.566434 mapread 0x07eea1 thru 0x080120 (0x1280 bytes) 000294[0] 1721406190.570265 mapwrite 0x3846ab thru 0x3860c8 (0x1a1e bytes) 000297[0] 1721406190.602718 trunc from 0x3860c9 to 0x41078b (0x8a6c3 bytes) 000298[0] 1721406190.607013 mapread 0x197bec thru 0x1998f8 (0x1d0d bytes) 000299[1] 1721406190.610628 trunc from 0x41078c to 0x76b8fc (0x35b171 bytes) 000301[0] 1721406190.627567 write 0x944a14 thru 0x945fee (0x15db bytes) 000302[1] 1721406190.635284 write 0x2925a6 thru 0x292fcf (0xa2a bytes) 000303[1] 1721406190.640204 write 0x58c587 thru 0x58e311 (0x1d8b bytes) 000304[1] 1721406190.654099 read 0x7dcfdd thru 0x7ddecb (0xeef bytes) 000305[0] 1721406190.658912 mapwrite 0x15d36b thru 0x15dd65 (0x9fb bytes) 000311[1] 1721406190.700361 trunc from 0x945fef to 0x5b184e (0x3947a0 bytes) 000312[1] 1721406190.708051 mapread 0x08c469 thru 0x08e1be (0x1d56 bytes) 000314[1] 1721406190.717062 mapread 0x3cfa62 thru 0x3d1744 (0x1ce3 bytes) 000315[1] 1721406190.724622 write 0x849146 thru 0x84a137 (0xff2 bytes) 000316[0] 1721406190.729565 mapread 0x4f0a1b thru 0x4f19b8 (0xf9e bytes) 000318[1] 1721406190.742960 trunc from 0x84a138 to 0x72576c (0x1249cb bytes) 000319[0] 1721406190.753044 read 0x5683d3 thru 0x56942b (0x1059 bytes) 000320[1] 1721406190.767313 mapwrite 0x1513df thru 0x152679 (0x129b bytes) 000322[0] 1721406190.793449 trunc from 0x72576d to 0x9c92e7 (0x2a3b7b bytes) 000324[1] 1721406190.806019 trunc from 0x9c92e8 to 0x913601 (0xb5ce6 bytes) 000325[1] 1721406190.813638 mapwrite 0x7be65d thru 0x7bf4ec (0xe90 bytes) 000326[1] 1721406190.829888 write 0x6db635 thru 0x6dbd1d (0x6e9 bytes) 000327[0] 1721406190.831623 trunc from 0x913602 to 0x775784 (0x19de7d bytes) 000329[0] 1721406190.847627 trunc from 0x775785 to 0x04a694 (0x72b0f0 bytes) 000331[0] 1721406190.857309 read 0x002686 thru 0x003467 (0xde2 bytes) 000332[0] 1721406190.864249 trunc from 0x04a695 to 0x49661f (0x44bf8b bytes) 000333[1] 1721406190.872512 write 0x8ce910 thru 0x8d0821 (0x1f12 bytes) 000334[1] 1721406190.880212 mapread 0x4ff207 thru 0x500f67 (0x1d61 bytes) 000335[1] 1721406190.885957 mapread 0x73d74d thru 0x73ef24 (0x17d8 bytes) 000336[0] 1721406190.891236 trunc from 0x8d0822 to 0x599089 (0x337798 bytes) 000338[1] 1721406190.904806 trunc from 0x59908a to 0x66348f (0xca406 bytes) 000341[1] 1721406190.920004 mapwrite 0x0ec4d2 thru 0x0ec93e (0x46d bytes) 000343[1] 1721406190.942628 mapread 0x63db6d thru 0x63f90b (0x1d9f bytes) 000344[0] 1721406190.946563 trunc from 0x663490 to 0x41a96e (0x248b21 bytes) 000346[0] 1721406190.959637 write 0x92de5b thru 0x92ed2e (0xed4 bytes) 000347[1] 1721406190.964444 read 0x149689 thru 0x149d14 (0x68c bytes) 000348[0] 1721406190.981024 read 0x2e6276 thru 0x2e7ab9 (0x1844 bytes) 000349[0] 1721406190.984767 read 0x3df674 thru 0x3e0107 (0xa94 bytes) 000352[0] 1721406191.012823 mapread 0x694d5f thru 0x69528e (0x530 bytes) 000355[1] 1721406191.016828 mapwrite 0x53226a thru 0x5329a3 (0x73a bytes) 000356[0] 1721406191.041436 mapwrite 0x8037a3 thru 0x80533d (0x1b9b bytes) 000357[0] 1721406191.065988 write 0x1fb69f thru 0x1fd1fd (0x1b5f bytes) 000359[1] 1721406191.077606 trunc from 0x92ed2f to 0x82f209 (0xffb25 bytes) 000362[0] 1721406191.096391 read 0x4e7f7e thru 0x4e90a1 (0x1124 bytes) 000363[0] 1721406191.104705 mapread 0x0cc0b7 thru 0x0cc0bd (0x7 bytes) 000364[1] 1721406191.113484 write 0x6ce4e5 thru 0x6cec19 (0x735 bytes) 000366[0] 1721406191.143358 trunc from 0x82f20a to 0x8e3f0f (0xb4d06 bytes) 000368[1] 1721406191.164620 trunc from 0x8e3f10 to 0x79b03f (0x148ed0 bytes) 000370[0] 1721406191.175718 write 0x783adb thru 0x784396 (0x8bc bytes) 000373[1] 1721406191.202278 read 0x0bb639 thru 0x0bcda2 (0x176a bytes) 000374[1] 1721406191.209907 trunc from 0x79b040 to 0x1e77c2 (0x5b387d bytes) 000375[0] 1721406191.219296 write 0x2f2752 thru 0x2f4244 (0x1af3 bytes) 000378[1] 1721406191.233304 mapwrite 0x77d574 thru 0x77f4c5 (0x1f52 bytes) 000380[0] 1721406191.287277 write 0x7f66de thru 0x7f8689 (0x1fac bytes) 000381[0] 1721406191.295909 mapwrite 0x794f6e thru 0x796ee7 (0x1f7a bytes) 000382[0] 1721406191.319092 trunc from 0x7f868a to 0x3a5a6e (0x452c1b bytes) 000383[1] 1721406191.332710 mapread 0x05bd9a thru 0x05d2a5 (0x150c bytes) 000384[0] 1721406191.343830 write 0x58c2a2 thru 0x58ca92 (0x7f1 bytes) 000386[1] 1721406191.350791 mapwrite 0x0953d0 thru 0x0954c3 (0xf4 bytes) 000387[0] 1721406191.374448 mapread 0x16f828 thru 0x1703d4 (0xbad bytes) 000388[0] 1721406191.381504 trunc from 0x58ca93 to 0x128def (0x463ca3 bytes) 000397[1] 1721406191.411481 read 0x01b905 thru 0x01c843 (0xf3f bytes) 000398[1] 1721406191.421786 write 0x3d3d24 thru 0x3d464f (0x92c bytes) 000399[0] 1721406191.427930 mapread 0x24a938 thru 0x24c8e5 (0x1fae bytes) 000400[0] 1721406191.440697 mapwrite 0x38394c thru 0x3857d2 (0x1e87 bytes) 000401[1] 1721406191.464327 mapread 0x34ddb4 thru 0x34fd38 (0x1f85 bytes) 000403[1] 1721406191.486324 write 0x159ceb thru 0x15ad0f (0x1025 bytes) 000404[0] 1721406191.495151 read 0x08c5a4 thru 0x08e446 (0x1ea3 bytes) 000407[0] 1721406191.500498 mapwrite 0x81cade thru 0x81d772 (0xc95 bytes) 000409[0] 1721406191.555433 mapread 0x0b2201 thru 0x0b3a23 (0x1823 bytes) 000410[1] 1721406191.559380 write 0x382543 thru 0x383614 (0x10d2 bytes) 000412[1] 1721406191.576425 trunc from 0x81d773 to 0x503302 (0x31a470 bytes) 000414[1] 1721406191.593828 mapwrite 0x61abaa thru 0x61ae81 (0x2d8 bytes) 000415[1] 1721406191.614243 write 0x44164b thru 0x442348 (0xcfe bytes) 000417[1] 1721406191.617025 mapread 0x309a24 thru 0x30b8a9 (0x1e86 bytes) 000420[1] 1721406191.628532 trunc from 0x61ae82 to 0x4a6320 (0x174b61 bytes) 000422[0] 1721406191.637704 mapwrite 0x9dcc45 thru 0x9de5a2 (0x195e bytes) 000423[0] 1721406191.672744 read 0x1839ee thru 0x184292 (0x8a5 bytes) 000424[0] 1721406191.711003 trunc from 0x9de5a3 to 0x93c746 (0xa1e5c bytes) 000425[1] 1721406191.715783 trunc from 0x93c747 to 0x91b38a (0x213bc bytes) 000426[0] 1721406191.730484 mapwrite 0x810b53 thru 0x810ee8 (0x396 bytes) 000431[0] 1721406191.766130 write 0x011cef thru 0x012d41 (0x1053 bytes) 000436[1] 1721406191.784947 trunc from 0x91b38b to 0x482d72 (0x498618 bytes) 000437[0] 1721406191.793189 trunc from 0x482d73 to 0x584ed0 (0x10215e bytes) 000438[0] 1721406191.804801 write 0x90452e thru 0x905d94 (0x1867 bytes) 000440[1] 1721406191.810681 write 0x68dbdb thru 0x68e512 (0x938 bytes) 000441[1] 1721406191.816308 trunc from 0x905d95 to 0x5082ce (0x3fdac6 bytes) 000442[1] 1721406191.827208 trunc from 0x5082cf to 0x49f8ce (0x68a00 bytes) 000443[0] 1721406191.834349 mapwrite 0x232608 thru 0x2342aa (0x1ca3 bytes) 000444[0] 1721406191.858911 write 0x478810 thru 0x478e33 (0x624 bytes) 000445[0] 1721406191.863571 write 0x8648b8 thru 0x865864 (0xfad bytes) 000451[0] 1721406191.884372 mapwrite 0x3778b3 thru 0x3793d6 (0x1b24 bytes) 000452[0] 1721406191.908564 mapwrite 0x97c640 thru 0x97d788 (0x1149 bytes) 000454[1] 1721406191.936563 mapwrite 0x89cdc9 thru 0x89e065 (0x129d bytes) 000455[1] 1721406191.961413 read 0x6f089d thru 0x6f138a (0xaee bytes) 000456[1] 1721406191.966735 read 0x2384a2 thru 0x238b08 (0x667 bytes) 000459[1] 1721406191.982625 mapwrite 0x257eaf thru 0x259dc7 (0x1f19 bytes) 000460[0] 1721406192.002673 mapread 0x826cb0 thru 0x827997 (0xce8 bytes) 000462[1] 1721406192.009544 trunc from 0x97d789 to 0x47418b (0x5095fd bytes) 000463[1] 1721406192.020532 read 0x0c72c8 thru 0x0c8b86 (0x18bf bytes) 000464[0] 1721406192.037963 write 0x7eaf11 thru 0x7ece28 (0x1f18 bytes) 000467[0] 1721406192.054029 read 0x3ecddd thru 0x3edd03 (0xf27 bytes) 000468[0] 1721406192.060031 write 0x3bc4d8 thru 0x3be376 (0x1e9f bytes) 000469[0] 1721406192.065549 trunc from 0x7ece29 to 0x62b1da (0x1c1c4e bytes) 000472[0] 1721406192.074384 trunc from 0x62b1db to 0x2bf95b (0x36b87f bytes) 000474[1] 1721406192.084540 trunc from 0x2bf95c to 0x03809d (0x2878be bytes) 000475[1] 1721406192.094110 read 0x02beb4 thru 0x02c38b (0x4d8 bytes) 000476[0] 1721406192.097501 mapwrite 0x08164b thru 0x083025 (0x19db bytes) 000478[0] 1721406192.130248 write 0x16eb5b thru 0x17046d (0x1913 bytes) 000479[0] 1721406192.134033 write 0x52444b thru 0x524e25 (0x9db bytes) 000481[1] 1721406192.136265 mapwrite 0x6508e7 thru 0x651d50 (0x146a bytes) 000482[1] 1721406192.183715 write 0x0dc62e thru 0x0dda28 (0x13fb bytes) 000483[0] 1721406192.185243 mapread 0x2330d9 thru 0x23486f (0x1797 bytes) 000484[0] 1721406192.203724 mapwrite 0x7a41be thru 0x7a5c58 (0x1a9b bytes) 000485[0] 1721406192.250248 mapread 0x1a0abc thru 0x1a1f55 (0x149a bytes) 000488[0] 1721406192.258025 write 0x659c42 thru 0x65b0a3 (0x1462 bytes) 000490[1] 1721406192.262419 write 0x7318cd thru 0x731bb6 (0x2ea bytes) 000492[0] 1721406192.272839 mapread 0x03c564 thru 0x03d916 (0x13b3 bytes) 000493[0] 1721406192.279453 read 0x0f7b4e thru 0x0f7c2d (0xe0 bytes) 000496[1] 1721406192.290664 read 0x292a58 thru 0x2934ec (0xa95 bytes) 000498[1] 1721406192.317185 mapread 0x001dfb thru 0x002304 (0x50a bytes) 000499[0] 1721406192.319649 mapwrite 0x7dea1f thru 0x7dfff2 (0x15d4 bytes) 000500[1] 1721406192.379474 mapread 0x3c88c9 thru 0x3c89f5 (0x12d bytes) All operations completed A-OK! PASS 16b (6s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 12:23:13 (1721406193) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 12:23:14 (1721406194) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0906794 s, 23.1 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0628372 s, 33.4 MB/s ldlm.namespaces.MGC192.168.203.140@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9cd2800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800b6e50800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a9cd2800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a9cd2800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.74691 s, 38.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.617532 s, 170 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.585204 s, 179 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.2311 s, 85.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.74699 s, 60.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.86854 s, 121 MB/s PASS 16d (10s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 12:23:24 (1721406204) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0588514 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0276713 s, 37.9 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 12:23:26 (1721406206) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 12:23:48 (1721406228) TIMEOUT PASS 16g (21s) == sanityn test 16i: read after truncate file ============ 12:24:09 (1721406249) ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (2s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 12:24:11 (1721406251) fail_loc=0x8000030a PASS 17 (3s) == sanityn test 18: mmap sanity check =========================================================================================== 12:24:14 (1721406254) mmap test1: basic mmap operation (PASS, 0.014148s) mmap test2: MAP_PRIVATE not write back (PASS, 0.010721s) mmap test3: concurrent mmap ops on two nodes (PASS, 2.2488s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 2.0578s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.021053s) mmap test6: check mmap write/read content on two nodes (PASS, 0.030498s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0109s) PASS 18 (8s) == sanityn test 19: test concurrent uncached read races ========================================================================= 12:24:22 (1721406262) oleg340-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 12:24:23 (1721406263) PASS 20 (1s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 12:24:24 (1721406264) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 12:24:26 (1721406266) now is 1721406266 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 new atime is 1721406327 PASS 23 (62s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 12:25:28 (1721406328) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.6K 369 488.3K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 376 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre[OST:1] filesystem_summary: 230.1K 369 229.8K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.6K 369 488.3K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 376 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 369 229.8K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 14.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500339 369 499970 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117976 376 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235633 369 235264 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 488.6K 369 488.3K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 376 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 115.3K 373 114.9K 1% /mnt/lustre2[OST:1] filesystem_summary: 230.1K 369 229.8K 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500339 369 499970 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 117976 376 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre[OST:1] filesystem_summary: 235633 369 235264 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 500339 369 499970 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 117976 376 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 118037 373 117664 1% /mnt/lustre2[OST:1] filesystem_summary: 235633 369 235264 1% /mnt/lustre2 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 14336 7524352 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 12:25:30 (1721406330) PASS 24b (1s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 12:25:31 (1721406331) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 12:25:33 (1721406333) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (0s) == sanityn test 26a: allow mtime to get older ============ 12:25:34 (1721406334) utime: good mknod times 1721406333 <= 1721406334 <= 1721406334 for /mnt/lustre/f26a utime: good mknod times 1721406333 <= 1721406334 <= 1721406334 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 12:25:36 (1721406336) PASS 26b (3s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 12:25:40 (1721406340) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 12:25:44 (1721406344) fail_loc=0x314 PASS 31a (2s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 12:25:46 (1721406346) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.203.140@tcp:/lustre 7542784 14336 7522304 1% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 12:26:03 (1721406363) -rw-r--r-- 1 root root 0 Jul 19 12:26 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: failed to access '/mnt/lustre2/f31r.sanityn.target': Input/output error PASS 31r (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 12:26:07 (1721406367) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (1s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-Lock-Cancel ========================================================== 12:26:08 (1721406368) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: DNE distributed operation should trigger COS ========================================================== 12:26:09 (1721406369) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: DNE local operation shouldn't trigger COS ========================================================== 12:26:10 (1721406370) SKIP: sanityn test_33e Need two or more clients, have 1 SKIP 33e (0s) == sanityn test 34: no lock timeout under IO ============= 12:26:10 (1721406370) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid in IDLE state after 0 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid in FULL state after 0 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid in IDLE state after 0 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid in FULL state after 0 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0000-osc-ffff8800a9cd2800.ost_server_uuid in IDLE state after 0 sec osc.lustre-OST0000-osc-ffff8800b6e50800.ost_server_uuid in IDLE state after 0 sec oleg340-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid,osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid 40 osc.lustre-OST0001-osc-ffff8800a9cd2800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0001-osc-ffff8800b6e50800.ost_server_uuid in IDLE state after 0 sec PASS 34 (71s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 12:27:21 (1721406441) fail_loc=0x80000317 Race attempt 0 Wait for 15476 15615 for 60 sec... - open/close 2712 (time 1721406452.77 total 10.00 last 271.16) total: 4000 open/close in 12.54 seconds: 318.97 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.203.140@tcp:/lustre 7.2G 14M 7.2G 1% /mnt/lustre 192.168.203.140@tcp:/lustre 7.2G 14M 7.2G 1% /mnt/lustre2 PASS 35 (63s) == sanityn test 36: handle ESTALE/open-unlink correctly == 12:28:24 (1721406504) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg340-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.299703 s, 175 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(0) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.295945 s, 177 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(1) *** before(7524352) after_dd(7473152) after(7524352) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.265731 s, 197 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(2) *** before(7524352) after_dd(7473152) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.295381 s, 177 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(3) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.281531 s, 186 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(4) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.30373 s, 173 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(5) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.266721 s, 197 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(6) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.298361 s, 176 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(7) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.297847 s, 176 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(8) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.27982 s, 187 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(9) *** before(7523328) after_dd(7472128) after(7523328) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.288609 s, 182 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(10) *** before(7523328) after_dd(7472128) after(7523328) PASS 36 (229s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 12:32:13 (1721406733) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 total: 10000 create in 9.46 seconds: 1057.04 ops/second PASS 37 (12s) == sanityn test 39a: test from 11063 ============================================================================================ 12:32:25 (1721406745) repeat after cancel_lru_locks PASS 39a (1s) == sanityn test 39b: 11063 problem 1 ============================================================================================ 12:32:27 (1721406747) repeat after cancel_lru_locks PASS 39b (2s) == sanityn test 39c: check truncate mtime update ================================================================================ 12:32:29 (1721406749) repeat after cancel_lru_locks PASS 39c (2s) == sanityn test 39d: sync write should update mtime ====== 12:32:32 (1721406752) fail_loc=0x411 fail_loc=0 PASS 39d (2s) == sanityn test 40a: pdirops: create vs others ======================================================================== 12:32:33 (1721406753) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (8s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 12:32:41 (1721406761) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (7s) == sanityn test 40c: pdirops: link and others ======================================================================== 12:32:48 (1721406768) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (8s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 12:32:56 (1721406776) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 12:33:03 (1721406783) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 12:33:10 (1721406790) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (4s) == sanityn test 41b: pdirops: create vs create ======================================================================== 12:33:14 (1721406794) open: File exists Conflict PASS 41b (4s) == sanityn test 41c: pdirops: create vs link ======================================================================== 12:33:18 (1721406798) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (4s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 12:33:22 (1721406802) Conflict PASS 41d (4s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 12:33:26 (1721406806) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 12:33:31 (1721406811) Conflict PASS 41f (4s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 12:33:35 (1721406815) Conflict PASS 41g (4s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 12:33:39 (1721406819) Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 12:33:44 (1721406824) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (657s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 12:44:41 (1721407481) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (4s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 12:44:45 (1721407485) open: File exists Conflict PASS 42b (4s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 12:44:49 (1721407489) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (5s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 12:44:54 (1721407494) Conflict PASS 42d (4s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 12:44:58 (1721407498) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (4s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 12:45:02 (1721407502) Conflict PASS 42f (4s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 12:45:06 (1721407506) Conflict PASS 42g (4s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 12:45:10 (1721407510) Conflict PASS 42h (4s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 12:45:14 (1721407514) PASS 43a (10s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 12:45:24 (1721407524) Conflict PASS 43b (4s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 12:45:28 (1721407528) Conflict PASS 43c (4s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 12:45:32 (1721407532) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (4s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 12:45:36 (1721407536) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 12:45:41 (1721407541) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (4s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 12:45:45 (1721407545) stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (4s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 12:45:49 (1721407549) Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 12:45:53 (1721407553) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 12:45:54 (1721407554) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (32s) == sanityn test 43k: unlink vs create ==================== 12:46:26 (1721407586) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (386s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 12:52:52 (1721407972) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Jul 19 12:52:55 EDT 2024 PASS 44a (4s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 12:52:56 (1721407976) open: File exists Conflict PASS 44b (5s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 12:53:01 (1721407981) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 12:53:07 (1721407987) Conflict PASS 44d (4s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 12:53:11 (1721407991) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 12:53:16 (1721407996) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 12:53:22 (1721408002) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 12:53:29 (1721408009) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 12:53:35 (1721408015) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 12:53:36 (1721408016) PASS 45a (47s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 12:54:23 (1721408063) Conflict PASS 45b (7s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 12:54:30 (1721408070) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 12:54:37 (1721408077) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (7s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 12:54:45 (1721408085) Conflict PASS 45e (8s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 12:54:53 (1721408093) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (8s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 12:55:01 (1721408101) stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (9s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 12:55:10 (1721408110) Conflict PASS 45h (7s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 12:55:17 (1721408117) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 12:55:18 (1721408118) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (600s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 13:05:18 (1721408718) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (5s) == sanityn test 46b: pdirops: link vs create ======================================================================== 13:05:23 (1721408723) open: File exists Conflict PASS 46b (5s) == sanityn test 46c: pdirops: link vs link ======================================================================== 13:05:29 (1721408729) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (5s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 13:05:33 (1721408733) Conflict PASS 46d (5s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 13:05:38 (1721408738) Conflict PASS 46e (5s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 13:05:43 (1721408743) Conflict PASS 46f (5s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 13:05:48 (1721408748) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 13:05:53 (1721408753) Conflict PASS 46h (5s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 13:05:58 (1721408758) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (0s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 13:05:59 (1721408759) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 13:05:59 (1721408759) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 13:06:00 (1721408760) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (1s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 13:06:01 (1721408761) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 13:06:02 (1721408762) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 13:06:03 (1721408763) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 13:06:04 (1721408764) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (0s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 13:06:04 (1721408764) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.00391196 s, 2.6 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 13:06:11 (1721408771) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.00283343 s, 56.1 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3175: kill: (24698) - No such process PASS 51a (3s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 13:06:14 (1721408774) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00274603 s, 373 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147165 s, 71.3 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 13:06:28 (1721408788) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00320291 s, 320 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 13:06:34 (1721408794) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113358 s, 92.5 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (3s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 13:06:37 (1721408797) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2438 0x986 0 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2438 0x986 0 read lease(1) has applied. PASS 51e (3s) == sanityn test 54: rename locking ======================= 13:06:40 (1721408800) llite.lustre-ffff8800a9cd2800.xattr_cache=1 llite.lustre-ffff8800b6e50800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504963 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:41.000000000 -0400 Modify: 2024-07-19 13:06:41.000000000 -0400 Change: 2024-07-19 13:06:41.000000000 -0400 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504966 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:46.000000000 -0400 Modify: 2024-07-19 13:06:46.000000000 -0400 Change: 2024-07-19 13:06:46.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504969 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:52.000000000 -0400 Modify: 2024-07-19 13:06:52.000000000 -0400 Change: 2024-07-19 13:06:52.000000000 -0400 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504968 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:52.000000000 -0400 Modify: 2024-07-19 13:06:52.000000000 -0400 Change: 2024-07-19 13:06:52.000000000 -0400 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504972 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:57.000000000 -0400 Modify: 2024-07-19 13:06:57.000000000 -0400 Change: 2024-07-19 13:06:57.000000000 -0400 Birth: - File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272504971 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:06:57.000000000 -0400 Modify: 2024-07-19 13:06:57.000000000 -0400 Change: 2024-07-19 13:06:57.000000000 -0400 Birth: - PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 13:07:04 (1721408824) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (6s) == sanityn test 55b: rename vs unlink source dir ========= 13:07:10 (1721408830) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 13:07:17 (1721408837) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 13:07:29 (1721408849) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (9s) == sanityn test 60: Verify data_version behaviour ======== 13:07:38 (1721408858) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 13:07:40 (1721408860) PASS 70a (1s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 13:07:41 (1721408861) PASS 70b (2s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 13:07:43 (1721408863) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00262763 s, 24.9 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 13:07:44 (1721408864) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00289195 s, 22.7 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 13:07:45 (1721408865) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (0s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 13:07:45 (1721408865) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00375158 s, 17.5 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 13:07:46 (1721408866) llite.lustre-ffff8800a9cd2800.xattr_cache=1 llite.lustre-ffff8800b6e50800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 13:07:48 (1721408868) llite.lustre-ffff8800a9cd2800.xattr_cache=1 llite.lustre-ffff8800b6e50800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (1s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 13:07:49 (1721408869) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00296486 s, 345 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00235892 s, 434 kB/s 3350: get lock2 3351: get lock1 3350: done 3350: get lock1 3351: done 3351: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 3350: cannot get lock1: Resource deadlock avoided fcntl cmd 7 failed: Resource deadlock avoided 3351: cannot get lock2: Resource deadlock avoided 3351: exit rc=0 3350: done 3350: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 13:07:55 (1721408875) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0129295 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00708599 s, 148 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3743: 3984 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (6s) == sanityn test 76: Verify MDT open_files listing ======== 13:08:01 (1721408881) opening files: ........ closing files: ........ PASS 76 (13s) == sanityn test 77a: check FIFO NRS policy =============== 13:08:14 (1721408894) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0971702 s, 173 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126792 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134564 s, 77.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128011 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149502 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173163 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103482 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106767 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0165631 s, 63.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0292082 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0248618 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284608 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370332 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0332929 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026943 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0216409 s, 48.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0301453 s, 34.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00771631 s, 136 MB/s 1048576 bytes (1.0 MB) copied, 0.00900601 s, 116 MB/s , 0.0098177 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0080764 s, 130 MB/s 1048576 bytes (1.0 MB) copied, 0.00986406 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104689 s, 100 MB/s 1048576 bytes (1.0 MB) copied, 0.00424222 s, 247 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00913236 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00738357 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00712872 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00731521 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00874999 s, 120 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00920607 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00140138 s, 748 MB/s 1048576 bytes (1.0 MB) copied, 0.0106072 s, 98.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00128918 s, 813 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=3 removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 13:08:16 (1721408896) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.10262 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00930288 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137405 s, 76.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189639 s, 55.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.021695 s, 48.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0167212 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198465 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023237 s, 45.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243964 s, 43.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0205634 s, 51.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394965 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0377931 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419148 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0318898 s, 32.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0415254 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0274446 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0338435 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00775524 s, 135 MB/s , 0.00824704 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00880112 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107443 s, 97.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00806864 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939672 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00938135 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0093449 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899905 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00160102 s, 655 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00158227 s, 663 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010277 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903149 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00985715 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0106086 s, 98.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00982816 s, 107 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0956089 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00877423 s, 120 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195041 s, 53.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196674 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0179978 s, 58.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208786 s, 50.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0194091 s, 54.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0205203 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146867 s, 71.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132248 s, 79.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0425142 s, 24.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0445652 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333914 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0425985 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0467327 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421534 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0524533 s, 20.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00801051 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00557607 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0060848 s, 172 MB/s 1048576 bytes (1.0 MB) copied, 0.00169218 s, 620 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00799969 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0071954 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00824153 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0078893 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00654483 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00124881 s, 840 MB/s 1048576 bytes (1.0 MB) copied, 0.00311157 s, 337 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857134 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00811805 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00201575 s, 520 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00115916 s, 905 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00130195 s, 805 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 13:08:19 (1721408899) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.115646 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015579 s, 67.3 MB/s 1048576 bytes (1.0 MB) copied, 0.015861 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0181928 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208485 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026565 s, 39.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106183 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.029589 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308627 s, 34.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0318175 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0349552 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0329619 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151352 s, 69.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227799 s, 46.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0199176 s, 52.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309422 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0170041 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00748852 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00950538 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00764401 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00620384 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00974976 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00784944 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00819077 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00876529 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00969337 s, 108 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00973125 s, 108 MB/s , 0.0101218 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112212 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114833 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106584 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119103 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118489 s, 88.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.10543 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115199 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185091 s, 56.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0366683 s, 28.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0307095 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305652 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0444029 s, 23.6 MB/s , 0.0357011 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410661 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0442852 s, 23.7 MB/s 1048576 bytes (1.0 MB) copied, 0.043535 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0486765 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0523931 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0545198 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.048247 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.04603 s, 22.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0526501 s, 19.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00869227 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00857091 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120357 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00875877 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00849093 s, 123 MB/s 1048576 bytes (1.0 MB) copied, 0.0116122 s, 90.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00905552 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00964003 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00946108 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117123 s, 89.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0121397 s, 86.4 MB/s , 0.0120556 s, 87.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130872 s, 80.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121876 s, 86.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0123221 s, 85.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0130141 s, 80.6 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0941175 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114279 s, 91.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126045 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0229325 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305339 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0445233 s, 23.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0491944 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0476407 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0501469 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0449648 s, 23.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0539266 s, 19.4 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0512383 s, 20.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0521507 s, 20.1 MB/s , 0.0542299 s, 19.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0486365 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0541116 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0475377 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00897667 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0090282 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00962082 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100843 s, 104 MB/s 1+0 records in 1+0 records out , 0.00997402 s, 105 MB/s 1048576 bytes (1.0 MB) copied, 0.0159794 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112398 s, 93.3 MB/s , 0.0110736 s, 94.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134925 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126488 s, 82.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0143428 s, 73.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130056 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143655 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148538 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133457 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131559 s, 79.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (4s) == sanityn test 77d: check TRR nrs policy ================ 13:08:23 (1721408903) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0940689 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129483 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0154214 s, 68.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130504 s, 80.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0200353 s, 52.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018504 s, 56.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208717 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0221057 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186258 s, 56.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218187 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419545 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271061 s, 38.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0352699 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224368 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0277296 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0416913 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390531 s, 26.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00484884 s, 216 MB/s , 0.00159894 s, 656 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00785445 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00865497 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00587068 s, 179 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124695 s, 84.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00762089 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00153987 s, 681 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00892414 s, 117 MB/s 1048576 bytes (1.0 MB) copied, 0.00985051 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00936362 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0094809 s, 111 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114138 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118625 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115838 s, 90.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00147534 s, 711 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.101078 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00794302 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120172 s, 87.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230062 s, 45.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0183501 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0298323 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250314 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251322 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0227814 s, 46.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260702 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186041 s, 56.4 MB/s 1048576 bytes (1.0 MB) copied, 0.019317 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293118 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258341 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195049 s, 53.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260669 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243369 s, 43.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0075143 s, 140 MB/s , 0.00908708 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00669929 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00724288 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010634 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00984481 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00767096 s, 137 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00160186 s, 655 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00124847 s, 840 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00871113 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00797114 s, 132 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0096267 s, 109 MB/s , 0.00135438 s, 774 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00223092 s, 470 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00155269 s, 675 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00122992 s, 853 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.132142 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00860617 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00938296 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0334323 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272436 s, 38.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316175 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373531 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0315929 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0293958 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0394313 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408002 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331722 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0428031 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359559 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0374377 s, 28.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0402752 s, 26.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0478669 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00642446 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00815317 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.001627 s, 644 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00755812 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895711 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00962237 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00892867 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00139227 s, 753 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100346 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00934467 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00175086 s, 599 MB/s 1048576 bytes (1.0 MB) copied, 0.00189271 s, 554 MB/s 1048576 bytes (1.0 MB) copied, 0.011788 s, 89.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00526676 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00150266 s, 698 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00202576 s, 518 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=3 removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 13:08:27 (1721408907) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.203.40@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.121538 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00786459 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142198 s, 73.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00673132 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0195724 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230249 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0192829 s, 54.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280809 s, 37.3 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0280749 s, 37.3 MB/s , 0.0138254 s, 75.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113516 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0258754 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152746 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122009 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00917053 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199707 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00751949 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00768662 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00558381 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00656932 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00811291 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00701629 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00653008 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00703695 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00802803 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00795246 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00768076 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00802831 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00865053 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00875638 s, 120 MB/s 1048576 bytes (1.0 MB) copied, 0.00853438 s, 123 MB/s , 0.00828525 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00869828 s, 121 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0808885 s, 207 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00870736 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121123 s, 86.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173341 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270304 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0271711 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03058 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294524 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246059 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262399 s, 40.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0333189 s, 31.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0364325 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0373794 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0313474 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.040077 s, 26.2 MB/s , 0.0319619 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0327934 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00693136 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00929512 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073513 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00748527 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0089862 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00983073 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00853043 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00834535 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100246 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00855407 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00972423 s, 108 MB/s , 0.00679798 s, 154 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00977011 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101071 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113029 s, 92.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126789 s, 82.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0980947 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00760173 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0089816 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100365 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193483 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316245 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300065 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0374679 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399514 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0419902 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0382615 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370416 s, 28.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0393804 s, 26.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0506182 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0390382 s, 26.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0457568 s, 22.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0461754 s, 22.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0083259 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00765481 s, 137 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00967932 s, 108 MB/s 1048576 bytes (1.0 MB) copied, 0.00596342 s, 176 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114811 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00915541 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00962142 s, 109 MB/s 1048576 bytes (1.0 MB) copied, 0.0102055 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00273497 s, 383 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00555857 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116674 s, 89.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109619 s, 95.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113842 s, 92.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00274237 s, 382 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00703701 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117651 s, 89.1 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.112957 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105536 s, 99.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119201 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294314 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171041 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270389 s, 38.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302264 s, 34.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0409761 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0395552 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0313391 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.042427 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0418598 s, 25.0 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=3 1048576 bytes (1.0 MB) copied, 0.0405565 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0388293 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044832 s, 23.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0488903 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0535791 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00751894 s, 139 MB/s , 0.00215406 s, 487 MB/s , 0.00554364 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00787288 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939029 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00365529 s, 287 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00597947 s, 175 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00141523 s, 741 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00136258 s, 770 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copiedldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=1 , 0.00127351 s, 823 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00812494 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00120421 s, 871 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00904551 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00127948 s, 820 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100191 s, 105 MB/s , 0.00938362 s, 112 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (6s) == sanityn test 77f: check TBF JobID nrs policy ========== 13:08:33 (1721408913) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0908174 s, 185 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00847522 s, 124 MB/s 1048576 bytes (1.0 MB) copied, 0.00873367 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0414027 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.04269 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.047018 s, 22.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0438122 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0495177 s, 21.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0543149 s, 19.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0524764 s, 20.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0533855 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0485304 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0547586 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0567841 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.055175 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0550714 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0578105 s, 18.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00589285 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00696111 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104977 s, 99.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00921078 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00378661 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00884192 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00800683 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00898591 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00977332 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0098349 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102256 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112863 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102737 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110504 s, 94.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111517 s, 94.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110108 s, 95.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.127993 s, 131 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00990832 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010374 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124536 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0396156 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0450333 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0385025 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0445119 s, 23.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0523966 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0525293 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0452291 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0430762 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0405971 s, 25.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.044313 s, 23.7 MB/s 1048576 bytes (1.0 MB) copied, 0.0494727 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0459886 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0490351 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110712 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113296 s, 92.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153511 s, 68.3 MB/s , 0.0115474 s, 90.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187955 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141184 s, 74.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01825 s, 57.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0177599 s, 59.0 MB/s , 0.0172139 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166535 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183674 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189291 s, 55.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193638 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0191294 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211642 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023312 s, 45.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.130835 s, 128 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137089 s, 76.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0141834 s, 73.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0225866 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0238506 s, 44.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309295 s, 33.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0248752 s, 42.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0326335 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267723 s, 39.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0250178 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263689 s, 39.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0293651 s, 35.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0245695 s, 42.7 MB/s , 0.0257834 s, 40.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0226716 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204953 s, 51.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132537 s, 79.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00957937 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00138298 s, 758 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00527233 s, 199 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00937702 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128989 s, 81.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011076 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129851 s, 80.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00152739 s, 687 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00163937 s, 640 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102522 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106347 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00146974 s, 713 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012384 s, 84.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00560903 s, 187 MB/s 1048576 bytes (1.0 MB) copied, 0.00476209 s, 220 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00501076 s, 209 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.087895 s, 191 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0081947 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122689 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107994 s, 97.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0135903 s, 77.2 MB/s , 0.0209345 s, 50.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246572 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0316901 s, 33.1 MB/s , 0.0249572 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171634 s, 61.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263898 s, 39.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273751 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375786 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.020563 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.023535 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230978 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0276679 s, 37.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717201 s, 146 MB/s , 0.00640482 s, 164 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00632002 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103352 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100732 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107065 s, 97.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00684763 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00873617 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0053171 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948011 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00685538 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00921506 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895562 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.009857 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00936628 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119441 s, 87.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8800b6e50800.lock_unused_count=3 removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (6s) == sanityn test 77g: Change TBF type directly ============ 13:08:39 (1721408919) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (2s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 13:08:41 (1721408921) oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 12 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 12 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 12 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 13:08:44 (1721408924) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 13:08:50 (1721408930) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0955669 s, 176 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00709409 s, 148 MB/s , 0.0071661 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178855 s, 58.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00942196 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171378 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224224 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178753 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188451 s, 55.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138731 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107858 s, 97.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.026839 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0246501 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0406061 s, 25.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0415918 s, 25.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0346386 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0328343 s, 31.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00714535 s, 147 MB/s , 0.0087853 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106656 s, 98.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823376 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012665 s, 82.8 MB/s 1048576 bytes (1.0 MB) copied, 0.00968896 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00994241 s, 105 MB/s , 0.0110755 s, 94.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101616 s, 103 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0103141 s, 102 MB/s 1048576 bytes (1.0 MB) copied, 0.00924014 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00852807 s, 123 MB/s 1048576 bytes (1.0 MB) copied, 0.0101367 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00823376 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135955 s, 77.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107136 s, 97.9 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.9367 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.428 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (46s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 13:09:36 (1721408976) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.641681 s, 26.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00853615 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224445 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0302464 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143913 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022254 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025398 s, 41.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0186492 s, 56.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193478 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.367808 s, 2.9 MB/s 1048576 bytes (1.0 MB) copied, 0.368437 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.376163 s, 2.8 MB/s , 0.37743 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.384972 s, 2.7 MB/s 1048576 bytes (1.0 MB) copied, 0.380496 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.377228 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.391579 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.150331 s, 7.0 MB/s , 0.151148 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151346 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.151398 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.15361 s, 6.8 MB/s , 0.15469 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152218 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153208 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152812 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152404 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153525 s, 6.8 MB/s , 0.154096 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153758 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153828 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.154544 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.156848 s, 6.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8288 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9694 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.646605 s, 25.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.168904 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.167245 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179633 s, 5.8 MB/s 1048576 bytes (1.0 MB) copied, 0.18031 s, 5.8 MB/s , 0.183469 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.192245 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.185692 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.192095 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.368783 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.38349 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.388044 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.378561 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.388036 s, 2.7 MB/s , 0.387092 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.398064 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.387028 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.15946 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.158832 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159112 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.157383 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159428 s, 6.6 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.158196 s, 6.6 MB/s , 0.158379 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159151 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159356 s, 6.6 MB/s , 0.158695 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.158925 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159724 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.160137 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.159864 s, 6.6 MB/s 1048576 bytes (1.0 MB) copied, 0.160251 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.161435 s, 6.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8279 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9666 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (132s) == sanityn test 77k: check TBF policy with NID/JobID/OPCode expression ========================================================== 13:11:48 (1721409108) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.192376 s, 87.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.017595 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0172672 s, 60.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0267154 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339428 s, 30.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0214995 s, 48.8 MB/s , 0.0277447 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0255424 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199893 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0709025 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0713067 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0873833 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0835352 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.096919 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0918545 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0956271 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10422 s, 10.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00848772 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00563981 s, 186 MB/s , 0.00548486 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00991379 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00993345 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00833626 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00446665 s, 235 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00752258 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108667 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00966269 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00896943 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102181 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00979764 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00983852 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100882 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105014 s, 99.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93645 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72954 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.203.40@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.203.40@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.189679 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118414 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125799 s, 83.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0354928 s, 29.5 MB/s , 0.0276899 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305018 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348963 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283575 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0297544 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0729685 s, 14.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0899616 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0925923 s, 11.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0886051 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.170336 s, 6.2 MB/s , 0.163227 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.257261 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.261984 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00748061 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00706626 s, 148 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00966705 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00658014 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00926313 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00779593 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00893264 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00804352 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935752 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00944727 s, 111 MB/s 1048576 bytes (1.0 MB) copied, 0.00934902 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00973393 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101389 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00913059 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00944183 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00923804 s, 114 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93651 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72608 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.203.40@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.193957 s, 86.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133043 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0233644 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0284816 s, 36.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173231 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0361278 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.031126 s, 33.7 MB/s 1048576 bytes (1.0 MB) copied, 0.030546 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279588 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0635759 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0622082 s, 16.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0740935 s, 14.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0850205 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0843418 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0870184 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0896484 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.096607 s, 10.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00783223 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0095156 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00904508 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00821714 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00932702 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00803321 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102425 s, 102 MB/s 1+0 records in 1+0 records out , 0.0082951 s, 126 MB/s 1048576 bytes (1.0 MB) copied, 0.010139 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00934522 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00996753 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106386 s, 98.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102434 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100228 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114823 s, 91.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108292 s, 96.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94219 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.87787 s, 21.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.343002 s, 48.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00846498 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114104 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112457 s, 93.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108881 s, 96.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012682 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00676488 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140092 s, 74.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00797087 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.176429 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.193473 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.190764 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.184367 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.369029 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.355831 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.381253 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.360321 s, 2.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00493135 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00941286 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113411 s, 92.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00818502 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00774468 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109925 s, 95.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00636138 s, 165 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00950703 s, 110 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0078623 s, 133 MB/s 1048576 bytes (1.0 MB) copied, 0.0101203 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00961181 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00930057 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117465 s, 89.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112267 s, 93.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013502 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115979 s, 90.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.77291 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72643 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.91612 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72841 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={1000} rate=5 running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.64281 s, 26.1 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00854583 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116733 s, 89.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228033 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0251323 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0233274 s, 45.0 MB/s , 0.00912976 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207243 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0168471 s, 62.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.361051 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.370032 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.369637 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.380519 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.380329 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.375743 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.364325 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.376818 s, 2.8 MB/s running as uid/gid/euid/egid 500/1000/500/1000, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00686301 s, 153 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0064589 s, 162 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0166371 s, 63.0 MB/s 1048576 bytes (1.0 MB) copied, 0.00704764 s, 149 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159948 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00751991 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0174441 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150635 s, 69.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00966593 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00987891 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00977263 s, 107 MB/s 1048576 bytes (1.0 MB) copied, 0.0101615 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00978513 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0095653 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00943941 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107398 s, 97.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8307 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/1000/500/1000, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4298 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.194041 s, 86.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122779 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0282079 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0301383 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.029868 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206295 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237542 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256005 s, 41.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0241061 s, 43.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0656032 s, 16.0 MB/s , 0.0623508 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0797237 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0816887 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0964267 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0973829 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0842551 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0915265 s, 11.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0093981 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01055 s, 99.4 MB/s , 0.00913163 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00860984 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00922375 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101243 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00843668 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00998917 s, 105 MB/s , 0.00997013 s, 105 MB/s , 0.00992827 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100783 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103557 s, 101 MB/s , 0.0101587 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101784 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100286 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104312 s, 101 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93833 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72717 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.340012 s, 49.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00796539 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00808756 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173553 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00775504 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212647 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126536 s, 82.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128641 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00633127 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.172395 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.184844 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.174638 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.191705 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.370179 s, 2.8 MB/s , 0.371783 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.370254 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.378585 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00600688 s, 175 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00732422 s, 143 MB/s , 0.00740648 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010468 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00784086 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00922789 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00985979 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0097388 s, 108 MB/s , 0.00955121 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00567527 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00725371 s, 145 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115052 s, 91.1 MB/s , 0.0103954 s, 101 MB/s , 0.00948029 s, 111 MB/s , 0.0113425 s, 92.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0142624 s, 73.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg340-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.77639 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7259 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.76951 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.72705 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (349s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:17:37 (1721409457) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:17:40 (1721409460) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg340-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.09937 s, 338 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg340-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0211 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (49s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:18:29 (1721409509) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0984369 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117331 s, 89.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0173738 s, 60.4 MB/s , 0.0161006 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0198476 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0256066 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0313025 s, 33.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0327223 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177581 s, 59.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0399945 s, 26.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283862 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252475 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0272007 s, 38.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0372449 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.033131 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0309423 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254571 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105327 s, 99.6 MB/s , 0.00769993 s, 136 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00946725 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00952468 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00883532 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00124639 s, 841 MB/s 1048576 bytes (1.0 MB) copied, 0.0115737 s, 90.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120445 s, 87.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00930086 s, 113 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8800b6e50800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114666 s, 91.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965675 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101966 s, 103 MB/s 1048576 bytes (1.0 MB) copied, 0.0111803 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00992999 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101705 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121541 s, 86.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.93746 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.96692 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.143407 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133994 s, 78.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316397 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0166609 s, 62.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207267 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0326057 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0283973 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0308233 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.022826 s, 45.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0683398 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0672434 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.164241 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.160333 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.261576 s, 4.0 MB/s , 0.267548 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.35764 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.35965 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0265935 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.025816 s, 40.6 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.0269845 s, 38.9 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0278786 s, 37.6 MB/s , 0.0287472 s, 36.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0282105 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0273809 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0275491 s, 38.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294771 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0289747 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0284647 s, 36.8 MB/s , 0.0295519 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306671 s, 34.2 MB/s , 0.0301033 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0300948 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0325457 s, 32.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg340-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg340-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94042 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.96527 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (63s) == sanityn test 77o: Changing rank should not panic ====== 13:19:32 (1721409572) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (3s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:19:35 (1721409575) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={500}&gid={500} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (25s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:20:00 (1721409600) mds.MDS.mdt.nrs_policies=tbf oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long pdsh@oleg340-client: oleg340-server: ssh exited with exit code 36 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument pdsh@oleg340-client: oleg340-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (10s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:20:10 (1721409610) oleg340-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device pdsh@oleg340-client: oleg340-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 13:20:13 (1721409613) File: '/mnt/lustre/d79.sanityn' Size: 512 Blocks: 1 IO Block: 131072 directory Device: 2c54f966h/743766374d Inode: 144115205272505431 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:20:13.000000000 -0400 Modify: 2024-07-19 13:20:13.000000000 -0400 Change: 2024-07-19 13:20:13.000000000 -0400 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (11s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:20:24 (1721409624) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:20:25 (1721409625) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:20:26 (1721409626) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:20:27 (1721409627) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (0s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:20:27 (1721409627) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:20:28 (1721409628) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.6837 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.6837 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:20:30 (1721409630) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (0s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:20:30 (1721409630) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:20:37 (1721409637) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 13:20:38 (1721409638) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:20:39 (1721409639) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (0s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:20:39 (1721409639) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2647 0xa57 0 1 2627 0xa43 0 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2628 0xa44 0 0 2648 0xa58 0 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 13:20:47 (1721409647) osc.lustre-OST0000-osc-ffff8800a9cd2800.idle_timeout=0 osc.lustre-OST0000-osc-ffff8800b6e50800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800a9cd2800.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800b6e50800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0428643 s, 191 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 2.99989 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5164: 4572 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0131144 s, 312 kB/s osc.lustre-OST0000-osc-ffff8800a9cd2800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6e50800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9cd2800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6e50800.idle_timeout=debug PASS 94 (10s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:20:57 (1721409657) llite.lustre-ffff8800a9cd2800.fast_read=0 llite.lustre-ffff8800b6e50800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00138828 s, 8.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800a9cd2800.fast_read=1 llite.lustre-ffff8800b6e50800.fast_read=1 PASS 95b (7s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:21:04 (1721409664) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:21:05 (1721409665) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00131896 s, 3.1 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:21:07 (1721409667) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0239562 s, 87.5 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (1s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:21:08 (1721409668) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0254294 s, 82.5 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 13:21:10 (1721409670) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00153943 s, 7.8 MB/s PASS 100e (1s) == sanityn test 101a: Discard DoM data on unlink ========= 13:21:11 (1721409671) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000488337 s, 8.4 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 13:21:13 (1721409673) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000462596 s, 8.9 MB/s PASS 101b (1s) == sanityn test 101c: Discard DoM data on close-unlink === 13:21:14 (1721409674) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000551968 s, 7.4 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 13:21:17 (1721409677) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1721409678 lockahead_test: lockahead_test.c:1186: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1721409680 Finishing test test23 at 1721409681 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:21:24 (1721409684) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 13:21:25 (1721409685) fail_loc=0x416 PASS 105 (47s) == sanityn test 106a: Verify the btime via statx() ======= 13:22:12 (1721409732) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (0s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:22:13 (1721409733) SKIP: sanityn test_106b statx() only test SKIP 106b (0s) == sanityn test 106c: Verify statx attributes mask ======= 13:22:13 (1721409733) SKIP: sanityn test_106c statx() only test SKIP 106c (1s) == sanityn test 107a: Basic grouplock conflict =========== 13:22:14 (1721409734) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0630159 s, 166 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2633 0xa49 0 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:22:18 (1721409738) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.0617232 s, 170 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2654 0xa5e 0 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6837 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (7s) == sanityn test 108a: lseek: parallel updates ============ 13:22:26 (1721409746) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.00123897 s, 8.1 MB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 4.12509 s, 2.0 MB/s PASS 108a (5s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:22:31 (1721409751) 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 5 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 9 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 13 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 16 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 22 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 25 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 26 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 28 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 29 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 30 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 32 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 35 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 36 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 39 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_size: failed to read coresize from /sys/module/acpi_cpufreq libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 49 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 192.168.203.140@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 192.168.203.140@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre (opts:) LNET ready to unload Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: detected 4 online CPUs by sysfs oleg340-server: Force libcfs to create 2 CPU partitions Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre Starting client: oleg340-client.virtnet: -o user_xattr,flock oleg340-server@tcp:/lustre /mnt/lustre2 PASS 109 (403s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:29:14 (1721410154) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-19 13:29:14.000000000 -0400 Modify: 2024-07-19 13:29:14.000000000 -0400 Change: 2024-07-19 13:29:14.000000000 -0400 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg340-client.virtnet oleg340-client.virtnet PASS 110 (36s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:29:50 (1721410190) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 13:29:51 (1721410191) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (0s) == sanityn test 113: check servers of specified fs ======= 13:29:51 (1721410191) lustre-OST0000-osc-ffff8800b623c800 active. lustre-OST0001-osc-ffff8800b623c800 active. lustre-OST0000-osc-ffff8800b623c800 active. lustre-OST0001-osc-ffff8800b623c800 active. MGC192.168.203.140@tcp active. PASS 113 (2s) cleanup: ====================================================== == sanityn test complete, duration 4224 sec ============== 13:29:53 (1721410193) Stopping clients: oleg340-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg340-client.virtnet /mnt/lustre2 opts: