-----============= acceptance-small: sanityn ============----- Mon Nov 11 12:28:01 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 27 28 102 skipping tests SLOW=no: 33a === sanityn: start setup 12:28:05 (1731346085) === Starting client oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Started clients oleg232-client.virtnet: 192.168.202.132@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg232-client.virtnet: executing check_config_client /mnt/lustre oleg232-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg232-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9dfc800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012aaf6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9dfc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aaf6000.idle_timeout=debug disable quota as required oleg232-server: oleg232-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 12:28:12 (1731346092) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109801 s, 95.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6953] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 12:28:13 (1731346093) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 12:28:15 (1731346095) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (1s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 12:28:16 (1731346096) -rw-r--r-- 1 root root 0 Nov 11 12:28 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 12:28:18 (1731346098) -rw-r--r-- 1 root root 0 Nov 11 12:28 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (1s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 12:28:19 (1731346099) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (1s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 12:28:21 (1731346101) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (1s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 12:28:22 (1731346102) total 1 -rwxrwxrwx 1 root root 0 Nov 11 12:28 f2a -rwxrwxrwx 1 root root 0 Nov 11 12:28 f2b total 1 -rwxrwxrwx 1 root root 0 Nov 11 12:28 f2a -rwxrwxrwx 1 root root 0 Nov 11 12:28 f2b running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00282639 s, 181 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 12:28:24 (1731346104) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 12:28:25 (1731346105) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 3.0333 s, 691 kB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (4s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 12:28:29 (1731346109) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 12:28:31 (1731346111) PASS 4 (2s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 12:28:33 (1731346113) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (2s) == sanityn test 6: remove of open file on other node ============================================================================ 12:28:35 (1731346115) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (1s) == sanityn test 7: remove of open directory on other node ======================================================================= 12:28:36 (1731346116) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (2s) == sanityn test 8: remove of open special file on other node ==================================================================== 12:28:38 (1731346118) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (1s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 12:28:39 (1731346119) PASS 9a (2s) == sanityn test 9b: append to striped sparse file ======== 12:28:41 (1731346121) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.01022 s, 103 MB/s 3+0 records in 3+0 records out 3 bytes (3 B) copied, 0.00855697 s, 0.4 kB/s Data read (expecting 'foo'): foo PASS 9b (1s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 12:28:42 (1731346122) 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00140682 s, 0.7 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00244566 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0137068 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00216243 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00269339 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0138729 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00262495 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00269759 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0130748 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00221191 s, 0.5 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.00272362 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte (1 B) copied, 0.0123091 s, 0.1 kB/s PASS 10a (2s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 12:28:44 (1731346124) 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.00263308 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00583708 s, 702 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB) copied, 0.000263952 s, 11.6 MB/s PASS 10b (1s) == sanityn test 11: execution of file opened for write should return error ============================================================== 12:28:45 (1731346125) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 301: /mnt/lustre2/d11/f: Text file busy PASS 11 (2s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 12:28:47 (1731346127) warning: '-runtime' deprecated, use '-t runtime' instead - link 3154 (time 1731346137.92 total 10.00 last 315.33) - link 6196 (time 1731346147.92 total 20.00 last 304.16) - link 9208 (time 1731346157.92 total 30.01 last 301.12) - link 10000 (time 1731346160.38 total 32.46 last 323.01) - link 13258 (time 1731346170.38 total 42.46 last 325.76) - link 16568 (time 1731346180.38 total 52.46 last 330.95) - link 19929 (time 1731346190.38 total 62.46 last 336.00) - link 23244 (time 1731346200.38 total 72.47 last 331.46) - link 26674 (time 1731346210.38 total 82.47 last 342.97) - link 30000 (time 1731346220.05 total 92.13 last 344.08) total: 32623 link in 100.00 seconds: 326.23 ops/second using seed 3341852948 running for 100 seconds - stat 10000 (time 1731346133 ; total 5 ; last 5) - stat 20000 (time 1731346138 ; total 10 ; last 5) - stat 30000 (time 1731346143 ; total 15 ; last 5) - stat 40000 (time 1731346147 ; total 19 ; last 4) - stat 50000 (time 1731346152 ; total 24 ; last 5) - stat 60000 (time 1731346157 ; total 29 ; last 5) - stat 70000 (time 1731346161 ; total 33 ; last 4) - stat 80000 (time 1731346165 ; total 37 ; last 4) - stat 90000 (time 1731346170 ; total 42 ; last 5) - stat 100000 (time 1731346174 ; total 46 ; last 4) - stat 110000 (time 1731346178 ; total 50 ; last 4) - stat 119752 (time 1731346189 ; total 61 ; last 11) - stat 120000 (time 1731346189 ; total 61 ; last 0) - stat 130000 (time 1731346199 ; total 71 ; last 10) - stat 139828 (time 1731346210 ; total 82 ; last 11) - stat 140000 (time 1731346210 ; total 82 ; last 0) - stat 150000 (time 1731346220 ; total 92 ; last 10) total: 157032 stats in 100 seconds: 1570.319946 stats/second - unlinked 0 (time 1731346178 ; total 0 ; last 0) - unlinked 10000 (time 1731346225 ; total 47 ; last 47) - unlinked 20000 (time 1731346249 ; total 71 ; last 24) - unlinked 30000 (time 1731346272 ; total 94 ; last 23) unlink(/mnt/lustre2/lockdir/lockfile32623) error: No such file or directory total: 32622 unlinks in 100 seconds: 326.220001 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21805) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21808) - No such process PASS 12 (152s) == sanityn test 13: test directory page revocation ======= 12:31:19 (1731346279) f13 f13 PASS 13 (1s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 12:31:20 (1731346280) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 336: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 23202 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 12:31:22 (1731346282) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4851: 23836 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (1s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 12:31:23 (1731346283) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 351: 24468 Terminated $DIR1/$tdir/sleep 60 PASS 14b (1s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 12:31:24 (1731346284) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 363: 25102 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 12:31:26 (1731346286) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 374: 25737 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 12:31:28 (1731346288) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_CODE=117440512 CLIENT_OS_VERSION_ID=7 STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424kB free gt MAXFREE 800000kB, increase 800000 (or reduce test fs size) to proceed sleep 5 for ZFS zfs Waiting for MDT destroys to complete SKIP 15 (11s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 12:31:39 (1731346299) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.31625 s, 26.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.286946 s, 29.2 MB/s Chance of close/open is 1 in 50 Seed set to 6301 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005[1] 1731346301.334608 write 0x48957b thru 0x497b16 (0xe59c bytes) truncating to largest ever: 0x349fcd 000007[0] 1731346301.340200 trunc from 0x497b17 to 0x349fcc (0x14db4a bytes) 000011[1] 1731346301.357373 write 0x6bc4c4 thru 0x6c92e1 (0xce1e bytes) 000012[1] 1731346301.374995 mapwrite 0x829180 thru 0x831e50 (0x8cd1 bytes) 000013[1] 1731346301.412870 mapread 0x508cde thru 0x517d74 (0xf097 bytes) 000014[1] 1731346301.424186 read 0x06547f thru 0x073bcc (0xe74e bytes) truncating to largest ever: 0x3fc2ab 000015[0] 1731346301.426990 trunc from 0x831e51 to 0x3fc2aa (0x435ba6 bytes) 000018[0] 1731346301.444204 read 0x233193 thru 0x23d027 (0x9e95 bytes) 000019[0] 1731346301.450126 read 0x32f3c5 thru 0x337518 (0x8154 bytes) 000020[1] 1731346301.476908 read 0x299941 thru 0x2a7d54 (0xe414 bytes) 000021[0] 1731346301.508136 trunc from 0x3fc2ab to 0x0c6101 (0x3361a9 bytes) 000022[0] 1731346301.522021 read 0x03596b thru 0x043d69 (0xe3ff bytes) 000023[1] 1731346301.531942 mapread 0x0594d7 thru 0x05ec30 (0x575a bytes) truncating to largest ever: 0x5c474f 000024[1] 1731346301.547210 trunc from 0x0c6102 to 0x5c474e (0x4fe64d bytes) 000027[1] 1731346301.556799 mapwrite 0x86d8ba thru 0x87033f (0x2a86 bytes) 000028[0] 1731346301.623991 mapwrite 0x53d8ea thru 0x54036b (0x2a82 bytes) 000032[1] 1731346301.691895 read 0x6c3fbe thru 0x6d2ced (0xed30 bytes) 000033[1] 1731346301.724520 read 0x4b1798 thru 0x4bae48 (0x96b1 bytes) truncating to largest ever: 0x862b93 000035[0] 1731346301.732829 trunc from 0x870340 to 0x862b92 (0xd7ad bytes) 000036[1] 1731346301.747984 read 0x0e54f4 thru 0x0f1792 (0xc29f bytes) 000037[1] 1731346301.777069 trunc from 0x862b93 to 0x732ab8 (0x1300da bytes) 000039[0] 1731346301.795353 read 0x701799 thru 0x701824 (0x8c bytes) 000041[0] 1731346301.803803 mapwrite 0x166d26 thru 0x173d5d (0xd038 bytes) 000043[0] 1731346301.853426 read 0x61981a thru 0x6253ee (0xbbd5 bytes) 000044[0] 1731346301.876094 read 0x28bcca thru 0x2975eb (0xb922 bytes) 000045[0] 1731346301.879886 trunc from 0x732ab9 to 0x1596c7 (0x5d93f1 bytes) 000048[1] 1731346301.898472 write 0x97e36a thru 0x982c00 (0x4897 bytes) 000049[0] 1731346301.908089 write 0x0e9d43 thru 0x0f9752 (0xfa10 bytes) 000053[0] 1731346301.942830 write 0x2d5cf4 thru 0x2de26c (0x8579 bytes) 000056[1] 1731346301.961715 mapread 0x4aa774 thru 0x4adf9b (0x3828 bytes) 000057[0] 1731346301.977912 mapread 0x3c379b thru 0x3d1a3e (0xe2a4 bytes) 000058[0] 1731346301.982063 mapread 0x1d31ea thru 0x1d444f (0x1266 bytes) 000060[1] 1731346301.985767 mapwrite 0x558f15 thru 0x560edf (0x7fcb bytes) 000062[1] 1731346302.057606 write 0x21d6df thru 0x21f9d3 (0x22f5 bytes) 000064[0] 1731346302.079375 trunc from 0x982c01 to 0x69b279 (0x2e7987 bytes) 000067[1] 1731346302.098733 read 0x4691ff thru 0x476aac (0xd8ae bytes) truncating to largest ever: 0x942ba0 000068[1] 1731346302.117173 trunc from 0x69b27a to 0x942b9f (0x2a7926 bytes) 000070[0] 1731346302.129300 write 0x2786c5 thru 0x280095 (0x79d1 bytes) 000071[1] 1731346302.137222 write 0x5f32d8 thru 0x5fec56 (0xb97f bytes) 000072[0] 1731346302.143159 mapread 0x69d641 thru 0x6ab806 (0xe1c6 bytes) 000076[1] 1731346302.179856 trunc from 0x942ba0 to 0x5deb0a (0x364095 bytes) 000077[0] 1731346302.193934 mapread 0x06f226 thru 0x078140 (0x8f1b bytes) 000083[0] 1731346302.219622 trunc from 0x5deb0b to 0x8809e5 (0x2a1edb bytes) 000084[0] 1731346302.231360 read 0x7d6c65 thru 0x7db36a (0x4706 bytes) 000089[1] 1731346302.273435 write 0x1aca8a thru 0x1af831 (0x2da8 bytes) 000091[0] 1731346302.310517 trunc from 0x8809e6 to 0x0ccd83 (0x7b3c62 bytes) 000092[1] 1731346302.325488 write 0x167e16 thru 0x17759c (0xf787 bytes) 000093[1] 1731346302.332446 mapwrite 0x475f5f thru 0x47cd5c (0x6dfe bytes) LOG DUMP (73 total operations): 1[0]: 1731346301.325418 SKIPPED (no operation) 2[0]: 1731346301.329681 SKIPPED (no operation) 3[0]: 1731346301.334324 SKIPPED (no operation) 4[1]: 1731346301.334608 WRITE 0x48957b thru 0x497b16 (0x0e59c bytes) HOLE 5[1]: 1731346301.339876 SKIPPED (no operation) 6[0]: 1731346301.340200 TRUNC_DN 0x497b17 to 0x349fcd 7[0]: 1731346301.352857 SKIPPED (no operation) 8[0]: 1731346301.355836 SKIPPED (no operation) 9[0]: 1731346301.357189 SKIPPED (no operation) 10[1]: 1731346301.357373 WRITE 0x6bc4c4 thru 0x6c92e1 (0x0ce1e bytes) HOLE 11[1]: 1731346301.374995 MAPWRITE 0x829180 thru 0x831e50 (0x08cd1 bytes) 12[1]: 1731346301.412870 MAPREAD 0x508cde thru 0x517d74 (0x0f097 bytes) 13[1]: 1731346301.424186 READ 0x6547f thru 0x73bcc (0x0e74e bytes) 14[0]: 1731346301.426990 TRUNC_DN 0x831e51 to 0x3fc2ab 15[0]: 1731346301.444204 READ 0x233193 thru 0x23d027 (0x09e95 bytes) 16[0]: 1731346301.450126 READ 0x32f3c5 thru 0x337518 (0x08154 bytes) 17[0]: 1731346301.476908 READ 0x299941 thru 0x2a7d54 (0x0e414 bytes) 18[0]: 1731346301.508136 TRUNC_DN 0x3fc2ab to 0xc6102 19[0]: 1731346301.522021 READ 0x3596b thru 0x43d69 (0x0e3ff bytes) 20[1]: 1731346301.531942 MAPREAD 0x594d7 thru 0x5ec30 (0x0575a bytes) 21[1]: 1731346301.547210 TRUNC_UP 0xc6102 to 0x5c474f 22[1]: 1731346301.556799 MAPWRITE 0x86d8ba thru 0x87033f (0x02a86 bytes) 23[0]: 1731346301.623991 MAPWRITE 0x53d8ea thru 0x54036b (0x02a82 bytes) 24[0]: 1731346301.688895 SKIPPED (no operation) 25[0]: 1731346301.691895 READ 0x6c3fbe thru 0x6d2ced (0x0ed30 bytes) 26[0]: 1731346301.724520 READ 0x4b1798 thru 0x4bae48 (0x096b1 bytes) 27[0]: 1731346301.732829 TRUNC_DN 0x870340 to 0x862b93 28[0]: 1731346301.747984 READ 0xe54f4 thru 0xf1792 (0x0c29f bytes) 29[1]: 1731346301.777069 TRUNC_DN 0x862b93 to 0x732ab9 30[1]: 1731346301.792129 SKIPPED (no operation) 31[1]: 1731346301.795353 READ 0x701799 thru 0x701824 (0x0008c bytes) 32[0]: 1731346301.803803 MAPWRITE 0x166d26 thru 0x173d5d (0x0d038 bytes) 33[0]: 1731346301.853426 READ 0x61981a thru 0x6253ee (0x0bbd5 bytes) 34[0]: 1731346301.876094 READ 0x28bcca thru 0x2975eb (0x0b922 bytes) 35[0]: 1731346301.879886 TRUNC_DN 0x732ab9 to 0x1596c8 36[0]: 1731346301.893065 SKIPPED (no operation) 37[1]: 1731346301.898472 WRITE 0x97e36a thru 0x982c00 (0x04897 bytes) HOLE 38[0]: 1731346301.908089 WRITE 0xe9d43 thru 0xf9752 (0x0fa10 bytes) 39[0]: 1731346301.932039 SKIPPED (no operation) 40[0]: 1731346301.936019 SKIPPED (no operation) 41[0]: 1731346301.939341 SKIPPED (no operation) 42[0]: 1731346301.942830 WRITE 0x2d5cf4 thru 0x2de26c (0x08579 bytes) 43[1]: 1731346301.961715 MAPREAD 0x4aa774 thru 0x4adf9b (0x03828 bytes) 44[0]: 1731346301.977912 MAPREAD 0x3c379b thru 0x3d1a3e (0x0e2a4 bytes) 45[0]: 1731346301.982063 MAPREAD 0x1d31ea thru 0x1d444f (0x01266 bytes) 46[0]: 1731346301.982494 SKIPPED (no operation) 47[1]: 1731346301.985767 MAPWRITE 0x558f15 thru 0x560edf (0x07fcb bytes) 48[1]: 1731346302.057606 WRITE 0x21d6df thru 0x21f9d3 (0x022f5 bytes) 49[1]: 1731346302.075790 SKIPPED (no operation) 50[0]: 1731346302.079375 TRUNC_DN 0x982c01 to 0x69b27a 51[0]: 1731346302.091964 SKIPPED (no operation) 52[0]: 1731346302.098733 READ 0x4691ff thru 0x476aac (0x0d8ae bytes) 53[1]: 1731346302.117173 TRUNC_UP 0x69b27a to 0x942ba0 54[1]: 1731346302.124985 SKIPPED (no operation) 55[0]: 1731346302.129300 WRITE 0x2786c5 thru 0x280095 (0x079d1 bytes) 56[1]: 1731346302.137222 WRITE 0x5f32d8 thru 0x5fec56 (0x0b97f bytes) 57[0]: 1731346302.143159 MAPREAD 0x69d641 thru 0x6ab806 (0x0e1c6 bytes) 58[0]: 1731346302.176443 SKIPPED (no operation) 59[0]: 1731346302.179668 SKIPPED (no operation) 60[1]: 1731346302.179856 TRUNC_DN 0x942ba0 to 0x5deb0b 61[0]: 1731346302.193934 MAPREAD 0x6f226 thru 0x78140 (0x08f1b bytes) 62[0]: 1731346302.208718 SKIPPED (no operation) 63[0]: 1731346302.216683 SKIPPED (no operation) 64[0]: 1731346302.219622 TRUNC_UP 0x5deb0b to 0x8809e6 65[0]: 1731346302.231360 READ 0x7d6c65 thru 0x7db36a (0x04706 bytes) 66[0]: 1731346302.271064 SKIPPED (no operation) 67[0]: 1731346302.272598 SKIPPED (no operation) 68[0]: 1731346302.272878 SKIPPED (no operation) 69[0]: 1731346302.273155 SKIPPED (no operation) 70[1]: 1731346302.273435 WRITE 0x1aca8a thru 0x1af831 (0x02da8 bytes) 71[0]: 1731346302.310517 TRUNC_DN 0x8809e6 to 0xccd84 72[1]: 1731346302.325488 WRITE 0x167e16 thru 0x17759c (0x0f787 bytes) HOLE 73[1]: 1731346302.332446 MAPWRITE 0x475f5f thru 0x47cd5c (0x06dfe bytes) Using seed 6301 Correct content saved for comparison (maybe hexdump "/mnt/lustre/f16a.sanityn" vs "/mnt/lustre/f16a.sanityn.fsxgood") sanityn test_16a: @@@@@@ FAIL: fsx failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7208:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:426:test_16a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7558:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7621:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7439:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:433:main() Dumping lctl log to /tmp/testlogs//sanityn.test_16a.*.1731346302.log rsync: chown "/tmp/testlogs/.sanityn.test_16a.debug_log.oleg232-server.1731346302.log.xhC530" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_16a.dmesg.oleg232-server.1731346302.log.ntUC0R" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 16a (5s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 12:31:45 (1731346305) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 18.7984 s, 446 kB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 9.92757 s, 845 kB/s Chance of close/open is 1 in 50 Seed set to 6334 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000005[0] 1731346334.856008 mapwrite 0x500114 thru 0x5006c8 (0x5b5 bytes) 000009[1] 1731346334.918556 mapwrite 0x134853 thru 0x1367b5 (0x1f63 bytes) 000010[1] 1731346334.969131 mapwrite 0x4f52c3 thru 0x4f61e1 (0xf1f bytes) 000011[0] 1731346335.012791 read 0x0ff6c8 thru 0x100261 (0xb9a bytes) 000012[1] 1731346335.017415 write 0x4f450c thru 0x4f516b (0xc60 bytes) 000013[0] 1731346335.023003 write 0x7488e3 thru 0x7491f4 (0x912 bytes) 000017[1] 1731346335.034167 read 0x3bdc66 thru 0x3becd0 (0x106b bytes) 000018[0] 1731346335.039021 mapwrite 0x90d57e thru 0x90d6d5 (0x158 bytes) LOG DUMP (12 total operations): 1[18446744073708284159]: 1731346334.851378 SKIPPED (no operation) 2[0]: 1731346334.856008 MAPWRITE 0x500114 thru 0x5006c8 (0x005b5 bytes) 3[0]: 1731346334.910341 SKIPPED (no operation) 4[1]: 1731346334.918556 MAPWRITE 0x134853 thru 0x1367b5 (0x01f63 bytes) 5[1]: 1731346334.969131 MAPWRITE 0x4f52c3 thru 0x4f61e1 (0x00f1f bytes) 6[1]: 1731346335.012791 READ 0xff6c8 thru 0x100261 (0x00b9a bytes) 7[1]: 1731346335.017415 WRITE 0x4f450c thru 0x4f516b (0x00c60 bytes) 8[0]: 1731346335.023003 WRITE 0x7488e3 thru 0x7491f4 (0x00912 bytes) HOLE 9[0]: 1731346335.030720 SKIPPED (no operation) 10[0]: 1731346335.030889 SKIPPED (no operation) 11[0]: 1731346335.034167 READ 0x3bdc66 thru 0x3becd0 (0x0106b bytes) 12[0]: 1731346335.039021 MAPWRITE 0x90d57e thru 0x90d6d5 (0x00158 bytes) Using seed 6334 Correct content saved for comparison (maybe hexdump "/mnt/lustre/f16b.sanityn" vs "/mnt/lustre/f16b.sanityn.fsxgood") sanityn test_16b: @@@@@@ FAIL: fsx with tiny write failed. Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7208:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:455:test_16b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7558:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7621:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7439:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:457:main() Dumping lctl log to /tmp/testlogs//sanityn.test_16b.*.1731346335.log rsync: chown "/tmp/testlogs/.sanityn.test_16b.debug_log.oleg232-server.1731346335.log.O2di0X" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_16b.dmesg.oleg232-server.1731346335.log.0me58b" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 16b (31s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 12:32:18 (1731346338) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 12:32:18 (1731346338) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.247936 s, 33.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB) copied, 0.256828 s, 32.7 MB/s ldlm.namespaces.MGC192.168.202.132@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a9dfc800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88012aaf6000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8800a9dfc800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8800a9dfc800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 2.459 s, 42.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 0.650548 s, 161 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.09088 s, 96.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.70417 s, 61.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.50953 s, 69.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 1.4256 s, 73.6 MB/s PASS 16d (12s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 12:32:31 (1731346351) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.110019 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184783 s, 56.7 MB/s PASS 16e (2s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 12:32:32 (1731346352) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 12:32:54 (1731346374) TIMEOUT PASS 16g (21s) == sanityn test 16h: mmap read after truncate file ======= 12:33:15 (1731346395) ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16h (2s) == sanityn test 16i: read after truncate file ============ 12:33:17 (1731346397) ------- before truncate -------- 0003220 3035 3a31 3035 3a31 2f3a 6f68 656d 732f 0003240 6e61 7469 7579 7273 3a31 622f 6e69 622f 0003260 7361 0a68 0003264 ------- after truncate -------- 0001460 7373 6468 783a 373a 3a34 3437 503a 6972 0001500 6976 656c 6567 732d 7065 7261 7461 6465 0001520 5320 4853 2f3a 6176 2f72 0001532 PASS 16i (1s) == sanityn test 16j: race dio with buffered i/o ========== 12:33:18 (1731346398) 1+0 records in 1+0 records out 26214400 bytes (26 MB) copied, 0.526106 s, 49.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.549522 s, 47.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.573504 s, 45.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.420524 s, 62.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.449554 s, 58.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.520505 s, 50.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.521396 s, 50.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.52873 s, 49.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.589815 s, 44.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.773641 s, 33.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.843084 s, 31.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.532251 s, 49.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.55478 s, 47.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.64263 s, 40.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.651926 s, 40.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.577144 s, 45.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.613557 s, 42.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.864347 s, 30.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB) copied, 0.884576 s, 29.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (8s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 12:33:26 (1731346406) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 LOG DUMP (25 total operations): 1: 1731346408.307001 SKIPPED (no operation) 2: 1731346408.308380 SKIPPED (no operation) 3: 1731346408.308606 SKIPPED (no operation) 4: 1731346408.309595 SKIPPED (no operation) 5: 1731346408.309727 MAPWRITE 0x0e8ca thru 0x173f7 (0x08b2e bytes) 6: 1731346408.356321 SKIPPED (no operation) 7: 1731346408.356452 READ 0x10925 thru 0x173f7 (0x06ad3 bytes) 8: 1731346408.356683 SKIPPED (no operation) 9: 1731346408.356811 MAPWRITE 0x5b513b thru 0x5c2b52 (0x0da18 bytes) 10: 1731346408.393761 READ 0x50cce4 thru 0x5156ef (0x08a0c bytes) 11: 1731346408.423987 WRITE 0xc1bcbe thru 0xc2332a (0x0766d bytes) HOLE 12: 1731346408.425858 SKIPPED (no operation) 13: 1731346408.426048 SKIPPED (no operation) 14: 1731346408.426240 MAPREAD 0x1d0bd0 thru 0x1df602 (0x0ea33 bytes) 15: 1731346408.426913 TRUNC_UP 0xc2332b to 0x1a7062c 16: 1731346408.433839 TRUNC_DN 0x1a7062c to 0x6d202c 17: 1731346408.436830 TRUNC_UP 0x6d202c to 0x232e1d3 18: 1731346408.452124 MAPREAD 0x4382cb thru 0x43bac6 (0x037fc bytes) 19: 1731346408.452505 WRITE 0x1963d thru 0x1e526 (0x04eea bytes) 20: 1731346408.453298 SKIPPED (no operation) 21: 1731346408.453477 READ 0x41390e thru 0x421f97 (0x0e68a bytes) 22: 1731346408.453780 TRUNC_DN 0x232e1d3 to 0xe58d39 23: 1731346408.459207 SKIPPED (no operation) 24: 1731346408.459428 SKIPPED (no operation) 25: 1731346408.459644 MAPWRITE 0x2e0d07 thru 0x2ed5ca (0x0c8c4 bytes) Using seed 1 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported LOG DUMP (5 total operations): 1: 1731346408.523880 SKIPPED (no operation) 2: 1731346408.524932 SKIPPED (no operation) 3: 1731346408.525127 SKIPPED (no operation) 4: 1731346408.527305 SKIPPED (no operation) 5: 1731346408.527456 MAPWRITE 0x0e8ca thru 0x173f7 (0x08b2e bytes) Using seed 1 skipping zero size read Correct content saved for comparison (maybe hexdump "/mnt/lustre/fsxfile.4" vs "/mnt/lustre/fsxfile.4.fsxgood") Correct content saved for comparison (maybe hexdump "/mnt/lustre/fsxfile.5" vs "/mnt/lustre/fsxfile.5.fsxgood") truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab LOG DUMP (629 total operations): 1: 1731346408.298135 SKIPPED (no operation) 2: 1731346408.299085 SKIPPED (no operation) 3: 1731346408.299310 SKIPPED (no operation) 4: 1731346408.300298 SKIPPED (no operation) 5: 1731346408.300427 MAPWRITE 0x0e8ca thru 0x173f7 (0x08b2e bytes) 6: 1731346408.346773 SKIPPED (no operation) 7: 1731346408.346879 READ 0x10925 thru 0x173f7 (0x06ad3 bytes) 8: 1731346408.347064 SKIPPED (no operation) 9: 1731346408.347180 MAPWRITE 0x5b513b thru 0x5c2b52 (0x0da18 bytes) 10: 1731346408.384485 READ 0x50cce4 thru 0x5156ef (0x08a0c bytes) 11: 1731346408.416086 WRITE 0xc1bcbe thru 0xc2332a (0x0766d bytes) HOLE 12: 1731346408.417998 SKIPPED (no operation) 13: 1731346408.418186 SKIPPED (no operation) 14: 1731346408.418384 MAPREAD 0x1d0bd0 thru 0x1df602 (0x0ea33 bytes) 15: 1731346408.419067 TRUNC_UP 0xc2332b to 0x1a7062c 16: 1731346408.434204 TRUNC_DN 0x1a7062c to 0x6d202c 17: 1731346408.438718 TRUNC_UP 0x6d202c to 0x232e1d3 18: 1731346408.454466 MAPREAD 0x4382cb thru 0x43bac6 (0x037fc bytes) 19: 1731346408.454795 WRITE 0x1963d thru 0x1e526 (0x04eea bytes) 20: 1731346408.455224 SKIPPED (no operation) 21: 1731346408.455345 READ 0x41390e thru 0x421f97 (0x0e68a bytes) 22: 1731346408.455522 TRUNC_DN 0x232e1d3 to 0xe58d39 23: 1731346408.460675 SKIPPED (no operation) 24: 1731346408.460969 SKIPPED (no operation) 25: 1731346408.461151 MAPWRITE 0x2e0d07 thru 0x2ed5ca (0x0c8c4 bytes) 26: 1731346408.491176 MAPREAD 0xd0bed8 thru 0xd178e3 (0x0ba0c bytes) 27: 1731346408.531953 SKIPPED (no operation) 28: 1731346408.532055 WRITE 0x34e2fd9 thru 0x34e5ce8 (0x02d10 bytes) HOLE 29: 1731346408.574627 WRITE 0x2f5195b thru 0x2f5a0eb (0x08791 bytes) 30: 1731346408.763516 WRITE 0x1cea66b thru 0x1cf2b36 (0x084cc bytes) 31: 1731346408.764200 WRITE 0x1d81f33 thru 0x1d8345d (0x0152b bytes) 32: 1731346408.764682 SKIPPED (no operation) 33: 1731346408.764864 MAPWRITE 0x24118b thru 0x250f34 (0x0fdaa bytes) 34: 1731346408.792210 SKIPPED (no operation) 35: 1731346408.792601 TRUNC_DN 0x34e5ce9 to 0xd72045 36: 1731346408.829206 SKIPPED (no operation) 37: 1731346408.829335 SKIPPED (no operation) 38: 1731346408.829493 READ 0x549c4c thru 0x54ab7a (0x00f2f bytes) 39: 1731346408.829712 TRUNC_UP 0xd72045 to 0x32cce75 40: 1731346408.839783 READ 0x547738 thru 0x54e23b (0x06b04 bytes) 41: 1731346408.840008 SKIPPED (no operation) 42: 1731346408.840116 WRITE 0x1fc80e0 thru 0x1fccc9c (0x04bbd bytes) 43: 1731346408.980663 MAPWRITE 0x1a1bcd2 thru 0x1a1e154 (0x02483 bytes) 44: 1731346409.044798 SKIPPED (no operation) 45: 1731346409.046635 MAPREAD 0x12ded65 thru 0x12e4ae7 (0x05d83 bytes) 46: 1731346409.046959 TRUNC_DN 0x32cce75 to 0x1bf9b05 47: 1731346409.095575 TRUNC_UP 0x1bf9b05 to 0x2922ad3 48: 1731346409.103723 SKIPPED (no operation) 49: 1731346409.103835 TRUNC_UP 0x2922ad3 to 0x2cdcba2 50: 1731346409.110909 TRUNC_DN 0x2cdcba2 to 0x65388b 51: 1731346409.147973 SKIPPED (no operation) 52: 1731346409.148149 MAPREAD 0x4ae5ac thru 0x4b21f1 (0x03c46 bytes) 53: 1731346409.148515 WRITE 0x394ecc9 thru 0x395c87a (0x0dbb2 bytes) HOLE 54: 1731346409.158429 WRITE 0x388df20 thru 0x389b0fd (0x0d1de bytes) 55: 1731346409.295403 SKIPPED (no operation) 56: 1731346409.295655 TRUNC_UP 0x395c87b to 0x3c569db 57: 1731346409.299827 TRUNC_DN 0x3c569db to 0x325434 58: 1731346409.311804 SKIPPED (no operation) 59: 1731346409.311965 SKIPPED (no operation) 60: 1731346409.312081 SKIPPED (no operation) 61: 1731346409.312325 READ 0x64080 thru 0x71e94 (0x0de15 bytes) 62: 1731346409.312910 READ 0x09849 thru 0x0f2af (0x05a67 bytes) 63: 1731346409.313070 MAPWRITE 0x374ccb6 thru 0x3751c84 (0x04fcf bytes) 64: 1731346409.788075 SKIPPED (no operation) 65: 1731346409.788299 SKIPPED (no operation) 66: 1731346409.788451 READ 0xd90cf7 thru 0xd99381 (0x0868b bytes) 67: 1731346409.788706 MAPWRITE 0x2dd8944 thru 0x2de38e9 (0x0afa6 bytes) 68: 1731346409.836282 SKIPPED (no operation) 69: 1731346409.836499 TRUNC_DN 0x3751c85 to 0x29fc22d 70: 1731346409.850879 READ 0x190a69f thru 0x190cebf (0x02821 bytes) 71: 1731346409.851184 TRUNC_DN 0x29fc22d to 0x972843 72: 1731346409.886485 MAPREAD 0x1f5297 thru 0x1fcdc3 (0x07b2d bytes) 73: 1731346409.886990 SKIPPED (no operation) 74: 1731346409.887411 SKIPPED (no operation) 75: 1731346409.887607 SKIPPED (no operation) 76: 1731346409.887807 MAPREAD 0x2151aa thru 0x21e70e (0x09565 bytes) 77: 1731346409.888179 TRUNC_DN 0x972843 to 0x152cc7 78: 1731346409.900146 SKIPPED (no operation) 79: 1731346409.900610 READ 0x124c5b thru 0x134451 (0x0f7f7 bytes) 80: 1731346409.900953 MAPWRITE 0x3cc5694 thru 0x3cd07c2 (0x0b12f bytes) 81: 1731346410.366955 READ 0x38666eb thru 0x3867537 (0x00e4d bytes) 82: 1731346410.367155 READ 0xe70859 thru 0xe750cb (0x04873 bytes) 83: 1731346410.367443 MAPREAD 0x3bed6a0 thru 0x3bf1afd (0x0445e bytes) 84: 1731346410.367698 MAPREAD 0x1a34c27 thru 0x1a43141 (0x0e51b bytes) 85: 1731346410.367971 SKIPPED (no operation) 86: 1731346410.368069 WRITE 0x1fcfa41 thru 0x1fd8a60 (0x09020 bytes) 87: 1731346410.368520 TRUNC_DN 0x3cd07c3 to 0x234b2f3 88: 1731346410.410162 MAPREAD 0x16c628 thru 0x1733c6 (0x06d9f bytes) 89: 1731346410.410799 SKIPPED (no operation) 90: 1731346410.410970 WRITE 0x9e2a02 thru 0x9ece3a (0x0a439 bytes) 91: 1731346410.411704 MAPREAD 0x1b42402 thru 0x1b460b2 (0x03cb1 bytes) 92: 1731346410.412007 SKIPPED (no operation) 93: 1731346410.412179 SKIPPED (no operation) 94: 1731346410.412564 SKIPPED (no operation) 95: 1731346410.412742 MAPWRITE 0x13a84c5 thru 0x13ab4f7 (0x03033 bytes) 96: 1731346410.444748 WRITE 0x43380a thru 0x43890f (0x05106 bytes) 97: 1731346410.541562 SKIPPED (no operation) 98: 1731346410.541678 SKIPPED (no operation) 99: 1731346410.541778 SKIPPED (no operation) 100: 1731346410.541879 MAPREAD 0x203ba8c thru 0x203fa8f (0x04004 bytes) 101: 1731346410.704772 WRITE 0xa0df33 thru 0xa0f950 (0x01a1e bytes) 102: 1731346410.720503 MAPREAD 0xd2a9a0 thru 0xd382bb (0x0d91c bytes) 103: 1731346410.737110 TRUNC_UP 0x234b2f3 to 0x33fb470 104: 1731346410.742746 SKIPPED (no operation) 105: 1731346410.742956 SKIPPED (no operation) 106: 1731346410.743152 SKIPPED (no operation) 107: 1731346410.743295 TRUNC_DN 0x33fb470 to 0x13fa054 108: 1731346410.750627 READ 0x72068c thru 0x723f93 (0x03908 bytes) 109: 1731346410.806718 MAPREAD 0xf6eef2 thru 0xf700a0 (0x011af bytes) 110: 1731346410.839565 MAPREAD 0x13c9de9 thru 0x13d36ef (0x09907 bytes) 111: 1731346410.860066 SKIPPED (no operation) 112: 1731346410.860219 READ 0x1018cd7 thru 0x1025774 (0x0ca9e bytes) 113: 1731346410.860490 WRITE 0x164c6ef thru 0x1651b7b (0x0548d bytes) HOLE 114: 1731346410.878851 TRUNC_UP 0x1651b7c to 0x3ba17c1 115: 1731346410.890101 SKIPPED (no operation) 116: 1731346410.890313 MAPWRITE 0x1b349d3 thru 0x1b3f9f6 (0x0b024 bytes) 117: 1731346411.054416 SKIPPED (no operation) 118: 1731346411.054525 TRUNC_DN 0x3ba17c1 to 0x2d6785d 119: 1731346411.174337 TRUNC_DN 0x2d6785d to 0x2373e10 120: 1731346411.189632 MAPWRITE 0x3e97e44 thru 0x3e9c106 (0x042c3 bytes) 121: 1731346411.294759 MAPREAD 0x899be1 thru 0x8a7879 (0x0dc99 bytes) 122: 1731346411.295205 MAPWRITE 0x1fe0dfc thru 0x1fe6bf1 (0x05df6 bytes) 123: 1731346411.349871 SKIPPED (no operation) 124: 1731346411.350065 SKIPPED (no operation) 125: 1731346411.350441 SKIPPED (no operation) 126: 1731346411.350919 READ 0x2b860ef thru 0x2b91870 (0x0b782 bytes) 127: 1731346411.425271 WRITE 0xdce7ed thru 0xddb0ac (0x0c8c0 bytes) 128: 1731346411.440746 MAPREAD 0x2d87241 thru 0x2d9241e (0x0b1de bytes) 129: 1731346411.445568 SKIPPED (no operation) 130: 1731346411.445750 MAPREAD 0x3058e42 thru 0x305ed48 (0x05f07 bytes) 131: 1731346411.449723 READ 0x2902505 thru 0x290aaf7 (0x085f3 bytes) 132: 1731346411.450075 SKIPPED (no operation) 133: 1731346411.450239 WRITE 0xff38c2 thru 0xff3c51 (0x00390 bytes) 134: 1731346411.481139 SKIPPED (no operation) 135: 1731346411.481358 SKIPPED (no operation) 136: 1731346411.482128 MAPWRITE 0x2271df1 thru 0x227a87d (0x08a8d bytes) 137: 1731346411.513946 SKIPPED (no operation) 138: 1731346411.514105 READ 0x26bf47 thru 0x26c2e7 (0x003a1 bytes) 139: 1731346411.514295 WRITE 0x1829462 thru 0x182e3bf (0x04f5e bytes) 140: 1731346411.538669 SKIPPED (no operation) 141: 1731346411.538784 SKIPPED (no operation) 142: 1731346411.539127 SKIPPED (no operation) 143: 1731346411.539317 TRUNC_DN 0x3e9c107 to 0x2135ab3 144: 1731346411.591484 WRITE 0x37709f5 thru 0x3772788 (0x01d94 bytes) HOLE 145: 1731346411.630915 SKIPPED (no operation) 146: 1731346411.631289 READ 0x26063f thru 0x2625a1 (0x01f63 bytes) 147: 1731346411.631705 SKIPPED (no operation) 148: 1731346411.631805 SKIPPED (no operation) 149: 1731346411.631906 TRUNC_DN 0x3772789 to 0x21ea83d 150: 1731346411.635492 SKIPPED (no operation) 151: 1731346411.635684 TRUNC_UP 0x21ea83d to 0x29b82cd 152: 1731346411.639915 SKIPPED (no operation) 153: 1731346411.640024 MAPWRITE 0x33f3c07 thru 0x33fc86f (0x08c69 bytes) 154: 1731346411.826064 SKIPPED (no operation) 155: 1731346411.826198 WRITE 0x2cd9f73 thru 0x2cded32 (0x04dc0 bytes) 156: 1731346411.835196 SKIPPED (no operation) 157: 1731346411.835427 MAPREAD 0xf56473 thru 0xf62b22 (0x0c6b0 bytes) 158: 1731346411.835822 MAPREAD 0x212f39e thru 0x213aedf (0x0bb42 bytes) 159: 1731346411.836057 MAPWRITE 0x1915d3e thru 0x1918368 (0x0262b bytes) 160: 1731346411.868057 SKIPPED (no operation) 161: 1731346411.868320 READ 0x2f4414a thru 0x2f4a602 (0x064b9 bytes) 162: 1731346411.868530 TRUNC_DN 0x33fc870 to 0xa4136a 163: 1731346411.907184 SKIPPED (no operation) 164: 1731346411.907458 TRUNC_UP 0xa4136a to 0x2584977 165: 1731346411.915976 WRITE 0x1665acc thru 0x16675d6 (0x01b0b bytes) 166: 1731346412.093458 SKIPPED (no operation) 167: 1731346412.093582 TRUNC_DN 0x2584977 to 0x24a0364 168: 1731346412.137145 SKIPPED (no operation) 169: 1731346412.137360 READ 0x10caf4d thru 0x10d3bd2 (0x08c86 bytes) 170: 1731346412.137657 SKIPPED (no operation) 171: 1731346412.137825 READ 0x87037 thru 0x8f917 (0x088e1 bytes) 172: 1731346412.138053 TRUNC_DN 0x24a0364 to 0xc9c2f1 173: 1731346412.166508 MAPWRITE 0x3b95d20 thru 0x3ba0da8 (0x0b089 bytes) 174: 1731346412.569798 MAPREAD 0x35eac71 thru 0x35eae17 (0x001a7 bytes) 175: 1731346412.570158 MAPWRITE 0x20f80 thru 0x2febc (0x0ef3d bytes) 176: 1731346412.612889 SKIPPED (no operation) 177: 1731346412.613058 MAPREAD 0x333a5a4 thru 0x333f84a (0x052a7 bytes) 178: 1731346412.613447 MAPREAD 0x2634f1 thru 0x26d405 (0x09f15 bytes) 179: 1731346412.613812 MAPREAD 0xac01c0 thru 0xacaed8 (0x0ad19 bytes) 180: 1731346412.614182 MAPWRITE 0x4c441 thru 0x53da6 (0x07966 bytes) 181: 1731346412.644839 MAPWRITE 0x3beae16 thru 0x3bf0f33 (0x0611e bytes) 182: 1731346412.679498 SKIPPED (no operation) 183: 1731346412.679852 SKIPPED (no operation) 184: 1731346412.679995 READ 0x16ea4cc thru 0x16eca9b (0x025d0 bytes) 185: 1731346412.680283 SKIPPED (no operation) 186: 1731346412.680455 MAPREAD 0x21efa1b thru 0x21fa9e8 (0x0afce bytes) 187: 1731346412.680823 READ 0x22e9a47 thru 0x22f3171 (0x0972b bytes) 188: 1731346412.681720 TRUNC_DN 0x3bf0f34 to 0x3bd993 189: 1731346412.733650 SKIPPED (no operation) 190: 1731346412.733845 TRUNC_UP 0x3bd993 to 0x2badcc3 191: 1731346412.743941 MAPWRITE 0x38745d7 thru 0x38764d0 (0x01efa bytes) 192: 1731346413.126690 MAPWRITE 0x2b77179 thru 0x2b7d1a4 (0x0602c bytes) 193: 1731346413.154634 MAPREAD 0xb17d0b thru 0xb1ba15 (0x03d0b bytes) 194: 1731346413.154915 SKIPPED (no operation) 195: 1731346413.155022 MAPREAD 0x1a9ba07 thru 0x1a9d47d (0x01a77 bytes) 196: 1731346413.155220 WRITE 0xc3f7f8 thru 0xc44988 (0x05191 bytes) 197: 1731346413.155804 TRUNC_DN 0x38764d1 to 0xd4cb4f 198: 1731346413.192020 SKIPPED (no operation) 199: 1731346413.192123 WRITE 0x27462df thru 0x275312d (0x0ce4f bytes) HOLE 200: 1731346413.197366 READ 0x1313171 thru 0x13229d3 (0x0f863 bytes) 201: 1731346413.275747 READ 0x0f454 thru 0x13ab4 (0x04661 bytes) 202: 1731346413.275920 SKIPPED (no operation) 203: 1731346413.276047 MAPREAD 0x1ad955e thru 0x1adb5b3 (0x02056 bytes) 204: 1731346413.326621 MAPREAD 0x1f1acbc thru 0x1f1de68 (0x031ad bytes) 205: 1731346413.378809 SKIPPED (no operation) 206: 1731346413.378970 READ 0xcf527 thru 0xdc587 (0x0d061 bytes) 207: 1731346413.379201 TRUNC_UP 0x275312e to 0x2b9b609 208: 1731346413.385547 SKIPPED (no operation) 209: 1731346413.385679 TRUNC_DN 0x2b9b609 to 0x22c4f0c 210: 1731346413.398789 WRITE 0x2f368a2 thru 0x2f38e7c (0x025db bytes) HOLE 211: 1731346413.414803 MAPREAD 0x65cc57 thru 0x662542 (0x058ec bytes) 212: 1731346413.415201 SKIPPED (no operation) 213: 1731346413.415402 SKIPPED (no operation) 214: 1731346413.415621 READ 0x1791782 thru 0x179ea9e (0x0d31d bytes) 215: 1731346413.416234 TRUNC_DN 0x2f38e7d to 0x2515e57 216: 1731346413.421115 SKIPPED (no operation) 217: 1731346413.421330 WRITE 0x14583ee thru 0x1464ef8 (0x0cb0b bytes) 218: 1731346413.422121 MAPREAD 0x24299a7 thru 0x242dc74 (0x042ce bytes) 219: 1731346413.439714 TRUNC_UP 0x2515e57 to 0x3c711db 220: 1731346413.447583 SKIPPED (no operation) 221: 1731346413.447737 SKIPPED (no operation) 222: 1731346413.447841 WRITE 0x675e1a thru 0x681e2a (0x0c011 bytes) 223: 1731346413.448471 SKIPPED (no operation) 224: 1731346413.448599 SKIPPED (no operation) 225: 1731346413.448699 SKIPPED (no operation) 226: 1731346413.448799 SKIPPED (no operation) 227: 1731346413.449070 WRITE 0x2e78036 thru 0x2e82333 (0x0a2fe bytes) 228: 1731346413.548694 MAPWRITE 0xb4c8ac thru 0xb5bc8a (0x0f3df bytes) 229: 1731346413.596087 READ 0x2c36aa0 thru 0x2c3c8c7 (0x05e28 bytes) 230: 1731346413.596411 SKIPPED (no operation) 231: 1731346413.596626 MAPREAD 0x1c2b8ce thru 0x1c35ce8 (0x0a41b bytes) 232: 1731346413.597015 MAPREAD 0x1cd03f8 thru 0x1cdd769 (0x0d372 bytes) 233: 1731346413.597450 READ 0x7ff5e1 thru 0x80bccd (0x0c6ed bytes) 234: 1731346413.597734 SKIPPED (no operation) 235: 1731346413.597835 READ 0x2433bc6 thru 0x24384fd (0x04938 bytes) 236: 1731346413.597981 SKIPPED (no operation) 237: 1731346413.598262 SKIPPED (no operation) 238: 1731346413.598444 SKIPPED (no operation) 239: 1731346413.598651 MAPREAD 0x5160f2 thru 0x51c6b9 (0x065c8 bytes) 240: 1731346413.599003 READ 0x26b3249 thru 0x26b3750 (0x00508 bytes) 241: 1731346413.599243 TRUNC_DN 0x3c711db to 0x32f323e 242: 1731346413.624036 MAPREAD 0x28e977b thru 0x28f3803 (0x0a089 bytes) 243: 1731346413.624542 MAPWRITE 0xa4abae thru 0xa4d118 (0x0256b bytes) 244: 1731346413.647723 WRITE 0x129dca thru 0x130c59 (0x06e90 bytes) 245: 1731346413.648243 SKIPPED (no operation) 246: 1731346413.648379 SKIPPED (no operation) 247: 1731346413.648479 SKIPPED (no operation) 248: 1731346413.648675 MAPWRITE 0x2edc1c3 thru 0x2ee57ce (0x0960c bytes) 249: 1731346413.698206 SKIPPED (no operation) 250: 1731346413.698368 MAPREAD 0x2907c1 thru 0x2940da (0x0391a bytes) 251: 1731346413.698674 TRUNC_DN 0x32f323e to 0x1bcecc1 252: 1731346413.725796 MAPREAD 0x96a404 thru 0x970277 (0x05e74 bytes) 253: 1731346413.726129 SKIPPED (no operation) 254: 1731346413.726243 SKIPPED (no operation) 255: 1731346413.726426 SKIPPED (no operation) 256: 1731346413.726733 SKIPPED (no operation) 257: 1731346413.726846 SKIPPED (no operation) 258: 1731346413.727036 WRITE 0x3486c9 thru 0x34cdda (0x04712 bytes) 259: 1731346413.742160 MAPREAD 0xb3f10c thru 0xb4649f (0x07394 bytes) 260: 1731346413.742697 READ 0xd35342 thru 0xd41428 (0x0c0e7 bytes) 261: 1731346413.742954 TRUNC_UP 0x1bcecc1 to 0x3a7d4f4 262: 1731346413.755377 MAPWRITE 0x12fbea1 thru 0x12fe9c9 (0x02b29 bytes) 263: 1731346413.781527 SKIPPED (no operation) 264: 1731346413.781647 WRITE 0x2ab0bca thru 0x2ab4e5d (0x04294 bytes) 265: 1731346413.882561 TRUNC_DN 0x3a7d4f4 to 0x235aab7 266: 1731346413.950554 SKIPPED (no operation) 267: 1731346413.950734 READ 0xd385f0 thru 0xd40278 (0x07c89 bytes) 268: 1731346414.041299 MAPWRITE 0xe4c76d thru 0xe57a8a (0x0b31e bytes) 269: 1731346414.215692 SKIPPED (no operation) 270: 1731346414.215905 SKIPPED (no operation) 271: 1731346414.216006 WRITE 0x23a8cc3 thru 0x23acb6d (0x03eab bytes) HOLE 272: 1731346414.216471 MAPREAD 0x1ea0278 thru 0x1ea9934 (0x096bd bytes) 273: 1731346414.217007 READ 0x2b135d thru 0x2b9913 (0x085b7 bytes) 274: 1731346414.217167 WRITE 0x2ad4418 thru 0x2adf63d (0x0b226 bytes) HOLE 275: 1731346414.219149 MAPWRITE 0x35f9e9 thru 0x36c254 (0x0c86c bytes) 276: 1731346414.248618 SKIPPED (no operation) 277: 1731346414.248729 WRITE 0x1faeb9c thru 0x1faf30f (0x00774 bytes) 278: 1731346414.257290 SKIPPED (no operation) 279: 1731346414.257474 SKIPPED (no operation) 280: 1731346414.257631 READ 0x100de63 thru 0x1015463 (0x07601 bytes) 281: 1731346414.257831 TRUNC_DN 0x2adf63e to 0x19ea9fb 282: 1731346414.269611 MAPWRITE 0x176c276 thru 0x177870a (0x0c495 bytes) 283: 1731346414.292171 MAPWRITE 0x17c39d1 thru 0x17c6db2 (0x033e2 bytes) 284: 1731346414.316547 SKIPPED (no operation) 285: 1731346414.316660 MAPWRITE 0x1d8f2dc thru 0x1d98f77 (0x09c9c bytes) 286: 1731346414.373639 TRUNC_DN 0x1d98f78 to 0x1683c92 287: 1731346414.387200 MAPWRITE 0x14cb712 thru 0x14d4e5d (0x0974c bytes) 288: 1731346414.413340 TRUNC_UP 0x1683c92 to 0x169e141 289: 1731346414.417978 SKIPPED (no operation) 290: 1731346414.418175 SKIPPED (no operation) 291: 1731346414.418319 SKIPPED (no operation) 292: 1731346414.418425 WRITE 0x274eb62 thru 0x2755205 (0x066a4 bytes) HOLE 293: 1731346414.422775 SKIPPED (no operation) 294: 1731346414.422983 SKIPPED (no operation) 295: 1731346414.423085 MAPWRITE 0x1a72b39 thru 0x1a7383e (0x00d06 bytes) 296: 1731346414.487545 MAPREAD 0x1c544c9 thru 0x1c55df4 (0x0192c bytes) 297: 1731346414.490313 TRUNC_UP 0x2755206 to 0x38e6c0d 298: 1731346414.498107 SKIPPED (no operation) 299: 1731346414.498274 TRUNC_UP 0x38e6c0d to 0x38fbd5c 300: 1731346414.501281 WRITE 0x2045565 thru 0x2055492 (0x0ff2e bytes) 301: 1731346414.511659 SKIPPED (no operation) 302: 1731346414.511872 MAPWRITE 0x1be370c thru 0x1bf337e (0x0fc73 bytes) 303: 1731346414.641290 WRITE 0x3d7ecf2 thru 0x3d80d36 (0x02045 bytes) HOLE 304: 1731346414.642588 READ 0x33885cc thru 0x338c526 (0x03f5b bytes) 305: 1731346414.642855 TRUNC_DN 0x3d80d37 to 0x39226a6 306: 1731346414.646010 MAPREAD 0xba8c6 thru 0xc6381 (0x0babc bytes) 307: 1731346414.646395 SKIPPED (no operation) 308: 1731346414.646547 TRUNC_DN 0x39226a6 to 0x1ff69d9 309: 1731346414.672729 SKIPPED (no operation) 310: 1731346414.673464 TRUNC_DN 0x1ff69d9 to 0x129af25 311: 1731346414.684382 READ 0x2db58b thru 0x2e0879 (0x052ef bytes) 312: 1731346414.684615 WRITE 0x43f7b thru 0x47928 (0x039ae bytes) 313: 1731346414.705409 SKIPPED (no operation) 314: 1731346414.705542 WRITE 0x2b9a97f thru 0x2ba9505 (0x0eb87 bytes) HOLE 315: 1731346414.717726 TRUNC_UP 0x2ba9506 to 0x3f0e56b 316: 1731346414.724227 READ 0x33450b9 thru 0x334d575 (0x084bd bytes) 317: 1731346414.954272 READ 0x36c7097 thru 0x36d6c6f (0x0fbd9 bytes) 318: 1731346414.954478 SKIPPED (no operation) 319: 1731346414.954604 MAPWRITE 0x24b51 thru 0x27c32 (0x030e2 bytes) 320: 1731346415.013963 SKIPPED (no operation) 321: 1731346415.014118 READ 0x10d5f54 thru 0x10db198 (0x05245 bytes) 322: 1731346415.014355 SKIPPED (no operation) 323: 1731346415.014586 WRITE 0x300074e thru 0x300b6b0 (0x0af63 bytes) 324: 1731346415.015118 READ 0x1b6f83c thru 0x1b78af6 (0x092bb bytes) 325: 1731346415.015313 SKIPPED (no operation) 326: 1731346415.015418 TRUNC_DN 0x3f0e56b to 0xb3115a 327: 1731346415.071209 MAPREAD 0x186452 thru 0x18a947 (0x044f6 bytes) 328: 1731346415.071637 SKIPPED (no operation) 329: 1731346415.071815 TRUNC_UP 0xb3115a to 0x313d487 330: 1731346415.086928 READ 0x248777b thru 0x248cc65 (0x054eb bytes) 331: 1731346415.230922 SKIPPED (no operation) 332: 1731346415.231022 SKIPPED (no operation) 333: 1731346415.231125 MAPREAD 0x304ee94 thru 0x3050360 (0x014cd bytes) 334: 1731346415.267962 TRUNC_DN 0x313d487 to 0x2ae4edd 335: 1731346415.278297 TRUNC_DN 0x2ae4edd to 0x19f8cbc 336: 1731346415.298478 MAPWRITE 0x2e3b8f1 thru 0x2e45a1e (0x0a12e bytes) 337: 1731346415.466142 WRITE 0xb0e7b5 thru 0xb1be19 (0x0d665 bytes) 338: 1731346415.466742 MAPREAD 0x1c8ab25 thru 0x1c8dd19 (0x031f5 bytes) 339: 1731346415.474768 SKIPPED (no operation) 340: 1731346415.475146 MAPWRITE 0x2df57dd thru 0x2df5913 (0x00137 bytes) 341: 1731346415.501805 WRITE 0x3635959 thru 0x3643b79 (0x0e221 bytes) HOLE 342: 1731346415.510969 WRITE 0xdb7437 thru 0xdc4089 (0x0cc53 bytes) 343: 1731346415.535677 SKIPPED (no operation) 344: 1731346415.535845 WRITE 0x20b446b thru 0x20bed34 (0x0a8ca bytes) 345: 1731346415.563884 SKIPPED (no operation) 346: 1731346415.564054 SKIPPED (no operation) 347: 1731346415.564278 MAPWRITE 0x212fafa thru 0x2136632 (0x06b39 bytes) 348: 1731346415.721795 SKIPPED (no operation) 349: 1731346415.722171 TRUNC_DN 0x3643b7a to 0xb8323b 350: 1731346415.745515 SKIPPED (no operation) 351: 1731346415.745655 SKIPPED (no operation) 352: 1731346415.745780 MAPWRITE 0x3e67420 thru 0x3e70d19 (0x098fa bytes) 353: 1731346415.861690 WRITE 0x1d11833 thru 0x1d1d49e (0x0bc6c bytes) 354: 1731346415.864757 READ 0xe6737 thru 0xe78dc (0x011a6 bytes) 355: 1731346415.864912 SKIPPED (no operation) 356: 1731346415.865450 SKIPPED (no operation) 357: 1731346415.865571 TRUNC_DN 0x3e70d1a to 0x1dd7007 358: 1731346415.880913 TRUNC_UP 0x1dd7007 to 0x222f1a3 359: 1731346415.885329 MAPREAD 0x72326d thru 0x726750 (0x034e4 bytes) 360: 1731346415.885895 MAPWRITE 0x11073e0 thru 0x11081e7 (0x00e08 bytes) 361: 1731346416.020904 SKIPPED (no operation) 362: 1731346416.021197 MAPREAD 0x1a0cd6a thru 0x1a0df6e (0x01205 bytes) 363: 1731346416.022170 SKIPPED (no operation) 364: 1731346416.022351 MAPREAD 0x58ed2 thru 0x5be4f (0x02f7e bytes) 365: 1731346416.024589 SKIPPED (no operation) 366: 1731346416.024787 MAPWRITE 0x3fffa51 thru 0x3ffffff (0x005af bytes) 367: 1731346416.119517 SKIPPED (no operation) 368: 1731346416.119732 SKIPPED (no operation) 369: 1731346416.119927 WRITE 0x2781f97 thru 0x278caae (0x0ab18 bytes) 370: 1731346416.178519 SKIPPED (no operation) 371: 1731346416.178719 WRITE 0xce669d thru 0xcf008c (0x099f0 bytes) 372: 1731346416.200777 TRUNC_DN 0x4000000 to 0x35e037e 373: 1731346416.236004 WRITE 0x138586b thru 0x138fbf4 (0x0a38a bytes) 374: 1731346416.245928 MAPWRITE 0x217f7d9 thru 0x2189dfb (0x0a623 bytes) 375: 1731346416.272796 MAPREAD 0x329f8e5 thru 0x32a63e7 (0x06b03 bytes) 376: 1731346416.273136 TRUNC_UP 0x35e037e to 0x3c97350 377: 1731346416.278589 READ 0xb073ba thru 0xb10e91 (0x09ad8 bytes) 378: 1731346416.279049 SKIPPED (no operation) 379: 1731346416.279217 TRUNC_DN 0x3c97350 to 0x1cafbcd 380: 1731346416.313810 SKIPPED (no operation) 381: 1731346416.313933 MAPREAD 0x183d4ae thru 0x184a028 (0x0cb7b bytes) 382: 1731346416.314303 SKIPPED (no operation) 383: 1731346416.314410 READ 0x1b9a8db thru 0x1b9b906 (0x0102c bytes) 384: 1731346416.314579 MAPWRITE 0xb40b9c thru 0xb472f9 (0x0675e bytes) 385: 1731346416.377783 TRUNC_DN 0x1cafbcd to 0x14d5182 386: 1731346416.389155 SKIPPED (no operation) 387: 1731346416.389369 MAPREAD 0x465f17 thru 0x469065 (0x0314f bytes) 388: 1731346416.389726 SKIPPED (no operation) 389: 1731346416.389889 WRITE 0x1f84f61 thru 0x1f8b444 (0x064e4 bytes) HOLE 390: 1731346416.394779 READ 0xf9cc43 thru 0xfac54c (0x0f90a bytes) 391: 1731346416.395010 SKIPPED (no operation) 392: 1731346416.395128 SKIPPED (no operation) 393: 1731346416.395239 WRITE 0x288cfdc thru 0x2894b71 (0x07b96 bytes) HOLE 394: 1731346416.397671 READ 0x257a3d thru 0x26386e (0x0be32 bytes) 395: 1731346416.398052 TRUNC_UP 0x2894b72 to 0x32d0b4d 396: 1731346416.406764 SKIPPED (no operation) 397: 1731346416.407160 SKIPPED (no operation) 398: 1731346416.407373 READ 0x1374c55 thru 0x1384530 (0x0f8dc bytes) 399: 1731346416.407788 TRUNC_DN 0x32d0b4d to 0x17cd279 400: 1731346416.416033 SKIPPED (no operation) 401: 1731346416.416354 WRITE 0x19fb56b thru 0x1a099a1 (0x0e437 bytes) HOLE 402: 1731346416.428184 MAPREAD 0xc9a65e thru 0xca6a29 (0x0c3cc bytes) 403: 1731346416.428619 READ 0x19445c6 thru 0x195330b (0x0ed46 bytes) 404: 1731346416.507906 SKIPPED (no operation) 405: 1731346416.508363 SKIPPED (no operation) 406: 1731346416.508506 TRUNC_UP 0x1a099a2 to 0x22a36e1 407: 1731346416.517036 WRITE 0x31ab914 thru 0x31b8d3e (0x0d42b bytes) HOLE 408: 1731346416.526526 SKIPPED (no operation) 409: 1731346416.526661 TRUNC_UP 0x31b8d3f to 0x372b4fe 410: 1731346416.533576 SKIPPED (no operation) 411: 1731346416.533688 SKIPPED (no operation) 412: 1731346416.533794 SKIPPED (no operation) 413: 1731346416.533891 SKIPPED (no operation) 414: 1731346416.534079 SKIPPED (no operation) 415: 1731346416.534186 MAPWRITE 0x3e0ca8e thru 0x3e16e16 (0x0a389 bytes) 416: 1731346416.757773 SKIPPED (no operation) 417: 1731346416.757993 READ 0x3b46d70 thru 0x3b52a08 (0x0bc99 bytes) 418: 1731346416.758269 TRUNC_DN 0x3e16e17 to 0x2adff8f 419: 1731346416.777018 READ 0x1ec013d thru 0x1ecac97 (0x0ab5b bytes) 420: 1731346416.777412 WRITE 0x133c4e3 thru 0x134545d (0x08f7b bytes) 421: 1731346416.787214 READ 0x1ba30c4 thru 0x1ba46d4 (0x01611 bytes) 422: 1731346416.787403 READ 0x265312f thru 0x2659b02 (0x069d4 bytes) 423: 1731346416.787590 TRUNC_DN 0x2adff8f to 0x19be0ab 424: 1731346416.808209 SKIPPED (no operation) 425: 1731346416.808457 MAPREAD 0xee0577 thru 0xee10ac (0x00b36 bytes) 426: 1731346416.809086 MAPWRITE 0x759215 thru 0x75d7de (0x045ca bytes) 427: 1731346416.828873 READ 0x2921ef thru 0x298c15 (0x06a27 bytes) 428: 1731346416.829138 TRUNC_DN 0x19be0ab to 0x112b56e 429: 1731346416.843695 SKIPPED (no operation) 430: 1731346416.843905 MAPREAD 0x30f755 thru 0x31ef73 (0x0f81f bytes) 431: 1731346416.844535 WRITE 0x342c5d thru 0x34ae8b (0x0822f bytes) 432: 1731346416.845052 MAPREAD 0x968cad thru 0x976255 (0x0d5a9 bytes) 433: 1731346416.845441 SKIPPED (no operation) 434: 1731346416.845659 SKIPPED (no operation) 435: 1731346416.846052 TRUNC_UP 0x112b56e to 0x212ad6d 436: 1731346416.852872 TRUNC_DN 0x212ad6d to 0xeb47d7 437: 1731346416.858771 SKIPPED (no operation) 438: 1731346416.858951 READ 0x38efe6 thru 0x39c42c (0x0d447 bytes) 439: 1731346416.859204 SKIPPED (no operation) 440: 1731346416.859389 SKIPPED (no operation) 441: 1731346416.859673 MAPWRITE 0x2c25991 thru 0x2c2b795 (0x05e05 bytes) 442: 1731346417.101879 READ 0xaea7bb thru 0xafa4d3 (0x0fd19 bytes) 443: 1731346417.187310 TRUNC_UP 0x2c2b796 to 0x30c882f 444: 1731346417.195639 TRUNC_UP 0x30c882f to 0x3dc8ce3 445: 1731346417.205294 MAPWRITE 0x38cb45e thru 0x38d2846 (0x073e9 bytes) 446: 1731346417.379821 READ 0xc99696 thru 0xc996a8 (0x00013 bytes) 447: 1731346417.379981 SKIPPED (no operation) 448: 1731346417.380129 SKIPPED (no operation) 449: 1731346417.380312 SKIPPED (no operation) 450: 1731346417.380595 SKIPPED (no operation) 451: 1731346417.380704 MAPWRITE 0x39f3309 thru 0x39fc1ce (0x08ec6 bytes) 452: 1731346417.431312 SKIPPED (no operation) 453: 1731346417.431425 TRUNC_DN 0x3dc8ce3 to 0x273553c 454: 1731346417.455649 READ 0x152dd7f thru 0x15311b6 (0x03438 bytes) 455: 1731346417.455871 MAPWRITE 0x108bea7 thru 0x1099c0c (0x0dd66 bytes) 456: 1731346417.491571 WRITE 0x96c6a2 thru 0x974555 (0x07eb4 bytes) 457: 1731346417.492454 WRITE 0x377694d thru 0x378003a (0x096ee bytes) HOLE 458: 1731346417.496675 SKIPPED (no operation) 459: 1731346417.496782 SKIPPED (no operation) 460: 1731346417.496878 WRITE 0x3f056a0 thru 0x3f141ed (0x0eb4e bytes) HOLE 461: 1731346417.515444 MAPWRITE 0x3c2cbc9 thru 0x3c3689c (0x09cd4 bytes) 462: 1731346417.668504 SKIPPED (no operation) 463: 1731346417.668672 MAPWRITE 0x3264306 thru 0x3269c6e (0x05969 bytes) 464: 1731346417.769279 MAPREAD 0xa36a98 thru 0xa43562 (0x0cacb bytes) 465: 1731346417.769633 WRITE 0x10595b5 thru 0x1059a64 (0x004b0 bytes) 466: 1731346417.769957 MAPREAD 0x762d9c thru 0x76dbb2 (0x0ae17 bytes) 467: 1731346417.770205 TRUNC_DN 0x3f141ee to 0x35e546b 468: 1731346417.780877 MAPREAD 0x28cb69b thru 0x28d92c1 (0x0dc27 bytes) 469: 1731346417.781299 SKIPPED (no operation) 470: 1731346417.781539 WRITE 0x2ef5c8 thru 0x2fddf6 (0x0e82f bytes) 471: 1731346417.792509 TRUNC_DN 0x35e546b to 0x201c265 472: 1731346417.817758 TRUNC_DN 0x201c265 to 0x1e5da6d 473: 1731346417.822359 READ 0x1535bf thru 0x15a19e (0x06be0 bytes) 474: 1731346417.822592 SKIPPED (no operation) 475: 1731346417.822696 WRITE 0x2e38475 thru 0x2e3e154 (0x05ce0 bytes) HOLE 476: 1731346417.826614 TRUNC_UP 0x2e3e155 to 0x346b2dd 477: 1731346417.831563 WRITE 0x2f4cda6 thru 0x2f52688 (0x058e3 bytes) 478: 1731346417.929521 SKIPPED (no operation) 479: 1731346417.929632 TRUNC_DN 0x346b2dd to 0xd7a8ab 480: 1731346417.982851 READ 0x772822 thru 0x7806e4 (0x0dec3 bytes) 481: 1731346417.983094 READ 0x942de2 thru 0x94668a (0x038a9 bytes) 482: 1731346417.983243 MAPWRITE 0x3f82838 thru 0x3f8e0bd (0x0b886 bytes) 483: 1731346418.403136 MAPREAD 0x58e869 thru 0x595d45 (0x074dd bytes) 484: 1731346418.403557 SKIPPED (no operation) 485: 1731346418.403703 SKIPPED (no operation) 486: 1731346418.403810 TRUNC_DN 0x3f8e0be to 0x2507b3c 487: 1731346418.428450 READ 0x19e2290 thru 0x19e73dd (0x0514e bytes) 488: 1731346418.428692 MAPWRITE 0x147e0fd thru 0x1486d9f (0x08ca3 bytes) 489: 1731346418.565931 SKIPPED (no operation) 490: 1731346418.566128 SKIPPED (no operation) 491: 1731346418.566352 READ 0x1fbea60 thru 0x1fbfc0e (0x011af bytes) 492: 1731346418.566658 READ 0x14227b8 thru 0x142a560 (0x07da9 bytes) 493: 1731346418.566942 SKIPPED (no operation) 494: 1731346418.567368 SKIPPED (no operation) 495: 1731346418.567580 SKIPPED (no operation) 496: 1731346418.567779 SKIPPED (no operation) 497: 1731346418.567972 SKIPPED (no operation) 498: 1731346418.568395 MAPWRITE 0x1f172d2 thru 0x1f1764f (0x0037e bytes) 499: 1731346418.585079 SKIPPED (no operation) 500: 1731346418.585535 READ 0xd367b9 thru 0xd3e171 (0x079b9 bytes) 501: 1731346418.585856 READ 0x42c113 thru 0x4384cc (0x0c3ba bytes) 502: 1731346418.586147 TRUNC_UP 0x2507b3c to 0x3c2fc0f 503: 1731346418.594921 READ 0x2cf21b8 thru 0x2cf2b41 (0x0098a bytes) 504: 1731346418.664107 TRUNC_DN 0x3c2fc0f to 0x1503d0d 505: 1731346418.720137 MAPREAD 0x13f5f7e thru 0x14015e0 (0x0b663 bytes) 506: 1731346418.720640 SKIPPED (no operation) 507: 1731346418.720938 TRUNC_DN 0x1503d0d to 0xc96242 508: 1731346418.738840 SKIPPED (no operation) 509: 1731346418.738963 MAPWRITE 0x1e6211f thru 0x1e67156 (0x05038 bytes) 510: 1731346418.848793 TRUNC_DN 0x1e67157 to 0x1b91038 511: 1731346418.857452 WRITE 0x328f66b thru 0x329944a (0x09de0 bytes) HOLE 512: 1731346418.863730 READ 0x30275a thru 0x311873 (0x0f11a bytes) 513: 1731346418.864047 MAPREAD 0x1af4651 thru 0x1b03117 (0x0eac7 bytes) 514: 1731346418.978446 MAPREAD 0x20c5d12 thru 0x20cec5d (0x08f4c bytes) 515: 1731346418.993713 SKIPPED (no operation) 516: 1731346418.993938 TRUNC_DN 0x329944b to 0x1d2ce5c 517: 1731346419.059217 TRUNC_DN 0x1d2ce5c to 0x1f9777 518: 1731346419.090830 WRITE 0x35627b thru 0x360079 (0x09dff bytes) HOLE 519: 1731346419.096085 TRUNC_UP 0x36007a to 0x2668008 520: 1731346419.109692 SKIPPED (no operation) 521: 1731346419.109969 MAPREAD 0x179f627 thru 0x17a887d (0x09257 bytes) 522: 1731346419.120608 TRUNC_UP 0x2668008 to 0x2c6d953 523: 1731346419.125827 SKIPPED (no operation) 524: 1731346419.126110 SKIPPED (no operation) 525: 1731346419.126236 READ 0x1115d17 thru 0x111f4f6 (0x097e0 bytes) 526: 1731346419.318543 SKIPPED (no operation) 527: 1731346419.318740 SKIPPED (no operation) 528: 1731346419.318919 MAPREAD 0xa822d6 thru 0xa8e755 (0x0c480 bytes) 529: 1731346419.319420 MAPWRITE 0xcf7605 thru 0xcf7c53 (0x0064f bytes) 530: 1731346419.350866 SKIPPED (no operation) 531: 1731346419.351070 SKIPPED (no operation) 532: 1731346419.351304 WRITE 0x2e86343 thru 0x2e93c27 (0x0d8e5 bytes) HOLE 533: 1731346419.402241 SKIPPED (no operation) 534: 1731346419.402364 TRUNC_DN 0x2e93c28 to 0xabff6f 535: 1731346419.441816 SKIPPED (no operation) 536: 1731346419.442400 MAPWRITE 0x9e8f89 thru 0x9ee4e5 (0x0555d bytes) 537: 1731346419.475182 SKIPPED (no operation) 538: 1731346419.475387 SKIPPED (no operation) 539: 1731346419.475551 MAPWRITE 0x2e400e9 thru 0x2e44cfa (0x04c12 bytes) 540: 1731346419.584289 READ 0x124f621 thru 0x125e5f3 (0x0efd3 bytes) 541: 1731346419.701150 READ 0xfd0987 thru 0xfdd782 (0x0cdfc bytes) 542: 1731346419.710878 SKIPPED (no operation) 543: 1731346419.711047 TRUNC_DN 0x2e44cfb to 0x1b6d5e2 544: 1731346419.792489 MAPREAD 0x153fa51 thru 0x15456a2 (0x05c52 bytes) 545: 1731346419.792828 SKIPPED (no operation) 546: 1731346419.792930 TRUNC_DN 0x1b6d5e2 to 0x10a7315 547: 1731346419.805620 WRITE 0x3b43922 thru 0x3b4fe75 (0x0c554 bytes) HOLE 548: 1731346419.817898 SKIPPED (no operation) 549: 1731346419.818310 TRUNC_DN 0x3b4fe76 to 0x16faff1 550: 1731346419.834692 WRITE 0x2bb1ed8 thru 0x2bb448d (0x025b6 bytes) HOLE 551: 1731346419.874877 TRUNC_UP 0x2bb448e to 0x37af169 552: 1731346419.887487 SKIPPED (no operation) 553: 1731346419.887661 MAPWRITE 0x19aa3e8 thru 0x19af709 (0x05322 bytes) 554: 1731346420.198379 TRUNC_UP 0x37af169 to 0x3f706ab 555: 1731346420.204792 READ 0xc113cc thru 0xc15039 (0x03c6e bytes) 556: 1731346420.318403 TRUNC_DN 0x3f706ab to 0x1f4a672 557: 1731346420.448672 MAPWRITE 0x1192b3a thru 0x119d327 (0x0a7ee bytes) 558: 1731346420.490381 TRUNC_DN 0x1f4a672 to 0xdd7d1 559: 1731346420.523722 MAPWRITE 0xe4bf4c thru 0xe53aa2 (0x07b57 bytes) 560: 1731346420.577508 MAPREAD 0xc29073 thru 0xc361ab (0x0d139 bytes) 561: 1731346420.707881 MAPWRITE 0x1173fc7 thru 0x117e8b0 (0x0a8ea bytes) 562: 1731346420.739750 MAPWRITE 0x1e2004e thru 0x1e29cad (0x09c60 bytes) 563: 1731346420.837712 MAPREAD 0xe4c70f thru 0xe50e94 (0x04786 bytes) 564: 1731346420.838025 SKIPPED (no operation) 565: 1731346420.838127 SKIPPED (no operation) 566: 1731346420.838226 MAPREAD 0x1ceafa0 thru 0x1cfadd6 (0x0fe37 bytes) 567: 1731346420.854932 MAPWRITE 0x2f82f16 thru 0x2f8f517 (0x0c602 bytes) 568: 1731346421.017075 WRITE 0x3a1377d thru 0x3a1d5ad (0x09e31 bytes) HOLE 569: 1731346421.020652 SKIPPED (no operation) 570: 1731346421.020850 TRUNC_DN 0x3a1d5ae to 0xc4cb9 571: 1731346421.025946 SKIPPED (no operation) 572: 1731346421.026163 MAPREAD 0x592f8 thru 0x60818 (0x07521 bytes) 573: 1731346421.026650 SKIPPED (no operation) 574: 1731346421.026825 MAPREAD 0x6aac7 thru 0x6fdbe (0x052f8 bytes) 575: 1731346421.027341 TRUNC_UP 0xc4cb9 to 0x1a97e95 576: 1731346421.037260 READ 0x187a935 thru 0x1883e7c (0x09548 bytes) 577: 1731346421.222772 READ 0xd2e8c2 thru 0xd3ae40 (0x0c57f bytes) 578: 1731346421.224628 READ 0x7f60fa thru 0x8024e7 (0x0c3ee bytes) 579: 1731346421.224908 READ 0xd91439 thru 0xd91bac (0x00774 bytes) 580: 1731346421.225088 MAPREAD 0x4feacb thru 0x50d9f9 (0x0ef2f bytes) 581: 1731346421.225692 MAPWRITE 0x380efed thru 0x381ae05 (0x0be19 bytes) 582: 1731346421.470134 SKIPPED (no operation) 583: 1731346421.470349 MAPREAD 0xc5d527 thru 0xc63713 (0x061ed bytes) 584: 1731346421.470740 MAPWRITE 0x1fe4efc thru 0x1fefb0d (0x0ac12 bytes) 585: 1731346421.521279 MAPWRITE 0x3af9d60 thru 0x3b08c42 (0x0eee3 bytes) 586: 1731346421.690897 MAPWRITE 0xea5c99 thru 0xeb5095 (0x0f3fd bytes) 587: 1731346421.727861 WRITE 0x30e25de thru 0x30edafd (0x0b520 bytes) 588: 1731346421.728660 SKIPPED (no operation) 589: 1731346421.728796 MAPWRITE 0x554ebb thru 0x5594e3 (0x04629 bytes) 590: 1731346421.746234 MAPWRITE 0x1bdbcb5 thru 0x1bddaec (0x01e38 bytes) 591: 1731346421.789234 MAPWRITE 0x3111a3f thru 0x3120906 (0x0eec8 bytes) 592: 1731346421.864628 READ 0x12a85d6 thru 0x12b433c (0x0bd67 bytes) 593: 1731346421.864968 SKIPPED (no operation) 594: 1731346421.865396 WRITE 0x272792b thru 0x2727fd0 (0x006a6 bytes) 595: 1731346421.885679 MAPREAD 0x3a2e56d thru 0x3a3c742 (0x0e1d6 bytes) 596: 1731346421.886196 SKIPPED (no operation) 597: 1731346421.886328 WRITE 0x3fc1a70 thru 0x3fc6da5 (0x05336 bytes) HOLE 598: 1731346421.911736 SKIPPED (no operation) 599: 1731346421.911901 TRUNC_DN 0x3fc6da6 to 0xf2f857 600: 1731346421.957740 MAPREAD 0x3eff51 thru 0x3f5924 (0x059d4 bytes) 601: 1731346421.958041 MAPWRITE 0x27474da thru 0x2755c51 (0x0e778 bytes) 602: 1731346422.115073 WRITE 0x348f57 thru 0x349c9c (0x00d46 bytes) 603: 1731346422.115407 SKIPPED (no operation) 604: 1731346422.115571 SKIPPED (no operation) 605: 1731346422.115769 READ 0x9c53a4 thru 0x9cf3fc (0x0a059 bytes) 606: 1731346422.116051 SKIPPED (no operation) 607: 1731346422.116159 MAPREAD 0x6d600a thru 0x6db14e (0x05145 bytes) 608: 1731346422.116401 TRUNC_DN 0x2755c52 to 0x1140bd 609: 1731346422.151720 READ 0xa556e thru 0xb1fb3 (0x0ca46 bytes) 610: 1731346422.152224 MAPREAD 0x383d0 thru 0x448a3 (0x0c4d4 bytes) 611: 1731346422.152707 TRUNC_UP 0x1140bd to 0x287d569 612: 1731346422.170452 READ 0xc24c86 thru 0xc2cdf4 (0x0816f bytes) 613: 1731346422.300783 MAPWRITE 0x1dfb0c thru 0x1e8dde (0x092d3 bytes) 614: 1731346422.372962 SKIPPED (no operation) 615: 1731346422.373168 MAPREAD 0xb9bc71 thru 0xba9f57 (0x0e2e7 bytes) 616: 1731346422.373543 MAPREAD 0x349a0c thru 0x3596b3 (0x0fca8 bytes) 617: 1731346422.373821 READ 0x151f2de thru 0x152ca15 (0x0d738 bytes) 618: 1731346422.374137 MAPREAD 0x8c6c7e thru 0x8cbf03 (0x05286 bytes) 619: 1731346422.374487 SKIPPED (no operation) 620: 1731346422.374848 MAPREAD 0x1b22bd2 thru 0x1b24911 (0x01d40 bytes) 621: 1731346422.375060 TRUNC_DN 0x287d569 to 0x2583160 622: 1731346422.399609 READ 0xcde7a thru 0xdbe8c (0x0e013 bytes) 623: 1731346422.400208 MAPREAD 0x71ea44 thru 0x71f4f2 (0x00aaf bytes) 624: 1731346422.400661 SKIPPED (no operation) 625: 1731346422.401013 WRITE 0x37de71f thru 0x37dff14 (0x017f6 bytes) HOLE 626: 1731346422.405739 SKIPPED (no operation) 627: 1731346422.405937 WRITE 0x37c6f04 thru 0x37cb810 (0x0490d bytes) 628: 1731346422.480895 READ 0xfe07f6 thru 0xfeb132 (0x0a93d bytes) 629: 1731346422.481205 MAPWRITE 0x328ab6 thru 0x32da35 (0x04f80 bytes) Using seed 1 Correct content saved for comparison (maybe hexdump "/mnt/lustre/fsxfile.2" vs "/mnt/lustre/fsxfile.2.fsxgood") All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Mon Nov 11 12:33:46 EST 2024 sanityn test_16k: @@@@@@ FAIL: - fsx 2 FAILURE! rc=1 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7208:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:858:test_16k() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7558:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7621:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7439:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:862:main() Dumping lctl log to /tmp/testlogs//sanityn.test_16k.*.1731346426.log rsync: chown "/tmp/testlogs/.sanityn.test_16k.debug_log.oleg232-server.1731346426.log.KlHfJ4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_16k.dmesg.oleg232-server.1731346426.log.nC0rza" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 16k (23s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 12:33:50 (1731346430) fail_loc=0x8000030a PASS 17 (4s) == sanityn test 18: mmap sanity check =========================================================================================== 12:33:54 (1731346434) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 879: 3212 Bus error $LUSTRE/tests/mmap_sanity -d $MOUNT1 -m $MOUNT2 $excepts sanityn test_18: @@@@@@ FAIL: mmap_sanity test failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7208:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:892:test_18() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7558:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7621:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7439:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:895:main() Dumping lctl log to /tmp/testlogs//sanityn.test_18.*.1731346435.log rsync: chown "/tmp/testlogs/.sanityn.test_18.debug_log.oleg232-server.1731346435.log.dt0CF8" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_18.dmesg.oleg232-server.1731346435.log.MMy29z" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2] FAIL 18 (2s) == sanityn test 19: test concurrent uncached read races ========================================================================= 12:33:57 (1731346437) oleg232-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 12:33:58 (1731346438) PASS 20 (2s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 12:34:00 (1731346440) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (1s) == sanityn test 23: others should see updated atime while another read============================================================== 12:34:01 (1731346441) now is 1731346442 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 new atime is 1731346503 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 12:35:04 (1731346504) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 64512 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3706880 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 126976 7411712 2% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 64512 3704832 2% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 62464 3706880 2% /mnt/lustre2[OST:1] filesystem_summary: 7542784 126976 7411712 2% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.7K 389 490.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 113.5K 407 113.1K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 113.6K 406 113.2K 1% /mnt/lustre[OST:1] filesystem_summary: 226.7K 389 226.3K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.7K 389 490.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 113.5K 407 113.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 113.6K 406 113.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 226.7K 389 226.3K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 63.0M 3.5G 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 61.0M 3.5G 2% /mnt/lustre[OST:1] filesystem_summary: 7.2G 124.0M 7.1G 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502514 389 502125 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 116247 407 115840 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 116310 406 115904 1% /mnt/lustre2[OST:1] filesystem_summary: 232133 389 231744 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 64512 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3706880 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 126976 7411712 2% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.7K 389 490.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 113.5K 407 113.1K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 113.6K 406 113.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 226.7K 389 226.3K 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502514 389 502125 1% /mnt/lustre[MDT:0] filesystem_summary: 502514 389 502125 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502514 389 502125 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 116247 407 115840 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 116310 406 115904 1% /mnt/lustre2[OST:1] filesystem_summary: 232133 389 231744 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 64512 3704832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 62464 3706880 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 126976 7411712 2% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 64512 3704832 2% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 62464 3706880 2% /mnt/lustre2[OST:1] filesystem_summary: 7542784 126976 7411712 2% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 12:35:06 (1731346506) PASS 24b (1s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 12:35:07 (1731346507) running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 12:35:09 (1731346509) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 12:35:10 (1731346510) utime: good mknod times 1731346510 <= 1731346511 <= 1731346511 for /mnt/lustre/f26a utime: good mknod times 1731346510 <= 1731346511 <= 1731346511 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 12:35:13 (1731346513) PASS 26b (3s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 12:35:17 (1731346517) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 File: '/mnt/lustre/f26c.sanityn' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502389 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2024-11-11 12:35:17.000000000 -0500 Birth: - File: '/mnt/lustre2/f26c.sanityn' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502389 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2024-11-11 12:35:17.000000000 -0500 Birth: - PASS 26c (2s) SKIP: sanityn test_27 skipping excluded test 27 SKIP: sanityn test_28 skipping ALWAYS excluded test 28 == sanityn test 30: recreate file race =================== 12:35:20 (1731346520) 0 PASS 30 (3s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 12:35:23 (1731346523) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 12:35:26 (1731346526) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.132@tcp:/lustre 7542784 126976 7409664 2% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 12:35:43 (1731346543) -rw-r--r-- 1 root root 0 Nov 11 12:35 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (4s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 12:35:47 (1731346547) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1213: 13665 Terminated tail -f $tfile PASS 31s (2s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 12:35:49 (1731346549) PASS 31t (2s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 12:35:52 (1731346552) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (0s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 12:35:52 (1731346552) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (1s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 12:35:53 (1731346553) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (1s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 12:35:54 (1731346554) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (0s) == sanityn test 34: no lock timeout under IO ============= 12:35:54 (1731346554) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid in FULL state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid in IDLE state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid in FULL state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid in IDLE state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a9dfc800.ost_server_uuid in FULL state after 0 sec oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8800a9dfc800.ost_server_uuid in IDLE state after 0 sec PASS 34 (100s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 12:37:34 (1731346654) fail_loc=0x80000317 Race attempt 0 Wait for 20269 20408 for 60 sec... - open/close 2679 (time 1731346665.84 total 10.00 last 267.79) total: 4000 open/close in 15.07 seconds: 265.37 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.132@tcp:/lustre 7.2G 125M 7.1G 2% /mnt/lustre 192.168.202.132@tcp:/lustre 7.2G 125M 7.1G 2% /mnt/lustre2 PASS 35 (64s) == sanityn test 36: handle ESTALE/open-unlink correctly == 12:38:38 (1731346718) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg232-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.882713 s, 59.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(0) *** before(7410688) after_dd(7357440) after(7408640) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.663311 s, 79.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(1) *** before(7408640) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.621819 s, 84.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(2) *** before(7410688) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.811953 s, 64.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(3) *** before(7410688) after_dd(7357440) after(7408640) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.803385 s, 65.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(4) *** before(7408640) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.790912 s, 66.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(5) *** before(7410688) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.731049 s, 71.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(6) *** before(7410688) after_dd(7357440) after(7408640) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.635724 s, 82.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(7) *** before(7408640) after_dd(7357440) after(7408640) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.54205 s, 96.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(8) *** before(7408640) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.538061 s, 97.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(9) *** before(7410688) after_dd(7359488) after(7410688) 50+0 records in 50+0 records out 52428800 bytes (52 MB) copied, 0.613225 s, 85.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete *** cycle(10) *** before(7410688) after_dd(7357440) after(7408640) PASS 36 (229s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 12:42:27 (1731346947) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 - create 9402 (time 1731346958.04 total 10.00 last 940.19) total: 10000 create in 10.61 seconds: 942.58 ops/second PASS 37 (13s) == sanityn test 39a: file mtime does not change after rename ========================================================== 12:42:40 (1731346960) repeat after cancel_lru_locks PASS 39a (1s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 12:42:41 (1731346961) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 12:42:44 (1731346964) repeat after cancel_lru_locks PASS 39c (3s) == sanityn test 39d: sync write should update mtime ====== 12:42:47 (1731346967) fail_loc=0x411 fail_loc=0 PASS 39d (1s) == sanityn test 40a: pdirops: create vs others ======================================================================== 12:42:48 (1731346968) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40a (7s) == sanityn test 40b: pdirops: open|create and others ======================================================================== 12:42:55 (1731346975) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (7s) == sanityn test 40c: pdirops: link and others ======================================================================== 12:43:03 (1731346983) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (7s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 12:43:09 (1731346989) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (7s) == sanityn test 40e: pdirops: rename and others ======================================================================== 12:43:16 (1731346996) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (7s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 12:43:23 (1731347003) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (4s) == sanityn test 41b: pdirops: create vs create ======================================================================== 12:43:27 (1731347007) open: File exists Conflict PASS 41b (4s) == sanityn test 41c: pdirops: create vs link ======================================================================== 12:43:31 (1731347011) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (5s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 12:43:36 (1731347016) Conflict PASS 41d (4s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 12:43:40 (1731347020) Conflict PASS 41e (4s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 12:43:44 (1731347024) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 12:43:49 (1731347029) Conflict PASS 41g (4s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 12:43:53 (1731347033) Conflict PASS 41h (4s) == sanityn test 41i: reint_open: create vs create ======== 12:43:57 (1731347037) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (647s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 12:54:44 (1731347684) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (4s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 12:54:48 (1731347688) open: File exists Conflict PASS 42b (4s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 12:54:52 (1731347692) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (4s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 12:54:56 (1731347696) Conflict PASS 42d (4s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 12:55:00 (1731347700) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (4s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 12:55:05 (1731347705) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 12:55:09 (1731347709) Conflict PASS 42g (4s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 12:55:13 (1731347713) Conflict PASS 42h (4s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 12:55:17 (1731347717) PASS 43a (12s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 12:55:29 (1731347729) Conflict PASS 43b (4s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 12:55:33 (1731347733) Conflict PASS 43c (4s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 12:55:37 (1731347737) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (4s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 12:55:41 (1731347741) Conflict PASS 43e (4s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 12:55:45 (1731347745) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 12:55:50 (1731347750) stat: cannot stat '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (4s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 12:55:54 (1731347754) Conflict PASS 43h (4s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 12:55:58 (1731347758) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 12:55:59 (1731347759) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (32s) == sanityn test 43k: unlink vs create ==================== 12:56:31 (1731347791) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (353s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 13:02:24 (1731348144) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Mon Nov 11 13:02:27 EST 2024 PASS 44a (4s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 13:02:28 (1731348148) open: File exists Conflict PASS 44b (4s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 13:02:32 (1731348152) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (4s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 13:02:36 (1731348156) Conflict PASS 44d (4s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 13:02:41 (1731348161) Conflict PASS 44e (5s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 13:02:45 (1731348165) Conflict PASS 44f (4s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 13:02:49 (1731348169) Conflict PASS 44g (4s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 13:02:53 (1731348173) Conflict PASS 44h (4s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 13:02:57 (1731348177) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 13:02:58 (1731348178) PASS 45a (22s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 13:03:20 (1731348200) Conflict PASS 45b (4s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 13:03:24 (1731348204) Conflict PASS 45c (4s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 13:03:29 (1731348209) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (5s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 13:03:33 (1731348213) Conflict PASS 45e (4s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 13:03:37 (1731348217) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (4s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 13:03:41 (1731348221) stat: cannot stat '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (4s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 13:03:45 (1731348225) Conflict PASS 45h (4s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 13:03:49 (1731348229) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 13:03:50 (1731348230) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (346s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 13:09:36 (1731348576) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (4s) == sanityn test 46b: pdirops: link vs create ======================================================================== 13:09:40 (1731348580) open: File exists Conflict PASS 46b (4s) == sanityn test 46c: pdirops: link vs link ======================================================================== 13:09:44 (1731348584) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (4s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 13:09:48 (1731348588) Conflict PASS 46d (4s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 13:09:52 (1731348592) Conflict PASS 46e (4s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 13:09:56 (1731348596) Conflict PASS 46f (4s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 13:10:00 (1731348600) Conflict PASS 46g (5s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 13:10:05 (1731348605) Conflict PASS 46h (4s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 13:10:09 (1731348609) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (0s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 13:10:09 (1731348609) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 13:10:10 (1731348610) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 13:10:11 (1731348611) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (0s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 13:10:11 (1731348611) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 13:10:12 (1731348612) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 13:10:13 (1731348613) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 13:10:14 (1731348614) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (0s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 13:10:14 (1731348614) 10+0 records in 10+0 records out 10240 bytes (10 kB) copied, 0.0162734 s, 629 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 13:10:21 (1731348621) 0+1 records in 0+1 records out 159 bytes (159 B) copied, 0.00295271 s, 53.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3518: kill: (31733) - No such process PASS 51a (3s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 13:10:24 (1731348624) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00206224 s, 497 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306487 s, 34.2 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 13:10:38 (1731348638) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00304154 s, 337 kB/s PASS 51c (6s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 13:10:44 (1731348644) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0243403 s, 43.1 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (2s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 13:10:46 (1731348646) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2448 0x990 0x280000400 read lease(1) has applied. PASS 51e (3s) == sanityn test 54: rename locking ======================= 13:10:49 (1731348649) llite.lustre-ffff8800a9dfc800.xattr_cache=1 llite.lustre-ffff88012aaf6000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505003 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:10:50.000000000 -0500 Modify: 2024-11-11 13:10:50.000000000 -0500 Change: 2024-11-11 13:10:50.000000000 -0500 Birth: - ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505006 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:10:55.000000000 -0500 Modify: 2024-11-11 13:10:55.000000000 -0500 Change: 2024-11-11 13:10:55.000000000 -0500 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: '/mnt/lustre/d1/d2' Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505009 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:11:01.000000000 -0500 Modify: 2024-11-11 13:11:01.000000000 -0500 Change: 2024-11-11 13:11:01.000000000 -0500 Birth: - open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505008 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:11:01.000000000 -0500 Modify: 2024-11-11 13:11:01.000000000 -0500 Change: 2024-11-11 13:11:01.000000000 -0500 Birth: - ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: '/mnt/lustre/d1' Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:11:06.000000000 -0500 Modify: 2024-11-11 13:11:06.000000000 -0500 Change: 2024-11-11 13:11:06.000000000 -0500 Birth: - File: '/mnt/lustre/d1/d2' Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505012 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:11:06.000000000 -0500 Modify: 2024-11-11 13:11:06.000000000 -0500 Change: 2024-11-11 13:11:06.000000000 -0500 Birth: - PASS 54 (23s) == sanityn test 55a: rename vs unlink target dir ========= 13:11:12 (1731348672) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (7s) == sanityn test 55b: rename vs unlink source dir ========= 13:11:19 (1731348679) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55b.sanityn/d1/d2' to '/mnt/lustre/d55b.sanityn/d3/d2': No such file or directory PASS 55b (7s) == sanityn test 55c: rename vs unlink orphan target dir == 13:11:26 (1731348686) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (12s) == sanityn test 55d: rename file vs link ================= 13:11:38 (1731348698) fail_loc=0x155 mv: cannot move '/mnt/lustre/d55d.sanityn/f1' to '/mnt/lustre/d55d.sanityn/d55d.sanityn': Is a directory PASS 55d (9s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 13:11:47 (1731348707) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1731348707 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.953735 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (65s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 13:12:52 (1731348772) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: read /mnt/lustre/f56b.sanityn@33554431995904+4096 no data: Success llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1731348773 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.519465 MB/s read offset: 32768000000kB -nan MB/s llverdev: data verified successfully PASS 56b (239s) == sanityn test 60: Verify data_version behaviour ======== 13:16:51 (1731349011) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 13:16:53 (1731349013) PASS 70a (1s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 13:16:54 (1731349014) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 13:16:57 (1731349017) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00222565 s, 29.4 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 13:16:58 (1731349018) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.0588768 s, 1.1 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 13:16:59 (1731349019) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (0s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 13:17:00 (1731349020) 1+0 records in 1+0 records out 65536 bytes (66 kB) copied, 0.00348166 s, 18.8 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (0s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 13:17:00 (1731349020) llite.lustre-ffff8800a9dfc800.xattr_cache=1 llite.lustre-ffff88012aaf6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 13:17:02 (1731349022) llite.lustre-ffff8800a9dfc800.xattr_cache=1 llite.lustre-ffff88012aaf6000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (1s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 13:17:03 (1731349023) 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00213171 s, 480 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00237974 s, 430 kB/s 11951: get lock2 11952: get lock1 11951: done 11951: get lock1 11952: done 11952: get lock2 fcntl cmd 7 failed: Resource deadlock avoided 11952: cannot get lock2: Resource deadlock avoided 11952: exit rc=0 11951: done 11951: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 13:17:09 (1731349029) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0549186 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00763619 s, 137 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4107: 12593 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (6s) == sanityn test 76: Verify MDT open_files listing ======== 13:17:15 (1731349035) opening files: ........ closing files: ........ PASS 76 (13s) == sanityn test 77a: check FIFO NRS policy =============== 13:17:28 (1731349048) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.173467 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073571 s, 143 MB/s , 0.00624165 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019119 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0410873 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260928 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0202384 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0495782 s, 21.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0419406 s, 25.0 MB/s , 0.0513909 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0525339 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0306531 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0340381 s, 30.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.091587 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.108931 s, 9.6 MB/s , 0.111125 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107831 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00709502 s, 148 MB/s 1048576 bytes (1.0 MB) copied, 0.00717841 s, 146 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0106484 s, 98.5 MB/s , 0.00961203 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00887224 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00866712 s, 121 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.010531 s, 99.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00956182 s, 110 MB/s , 0.00886856 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00899712 s, 117 MB/s 1048576 bytes (1.0 MB) copied, 0.012215 s, 85.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107714 s, 97.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00986226 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101979 s, 103 MB/s , 0.0108313 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108553 s, 96.6 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77a.sanityn' PASS 77a (2s) == sanityn test 77b: check CRR-N NRS policy ============== 13:17:30 (1731349050) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.174596 s, 96.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00717659 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903605 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331657 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0387468 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339672 s, 30.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=3 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.053772 s, 19.5 MB/s , 0.0654257 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103796 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0101521 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133431 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00935061 s, 112 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00948236 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0097837 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00990514 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105829 s, 99.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0096183 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00964174 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103664 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107284 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123341 s, 85.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112837 s, 92.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108483 s, 96.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130809 s, 80.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0784538 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0952529 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0911192 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.093991 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0922169 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0907333 s, 11.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.118967 s, 8.8 MB/s , 0.114227 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113931 s, 9.2 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.118046 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110612 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0169594 s, 61.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03678 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0294957 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0393953 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408577 s, 25.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0471951 s, 22.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0423644 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0746916 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00200627 s, 523 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00134852 s, 778 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206547 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00183974 s, 570 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0260087 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178279 s, 58.8 MB/s , 0.0249372 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00185855 s, 564 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0301493 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019776 s, 53.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140326 s, 74.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132065 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124041 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.086012 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.016063 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00914976 s, 115 MB/s 1048576 bytes (1.0 MB) copied, 0.00839179 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0821728 s, 12.8 MB/s 1048576 bytes (1.0 MB) copied, 0.103899 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.100555 s, 10.4 MB/s 1+0 records in 1+0 records out , 0.120858 s, 8.7 MB/s 1048576 bytes (1.0 MB) copied, 0.10978 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.129772 s, 8.1 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (3s) == sanityn test 77c: check ORR NRS policy ================ 13:17:33 (1731349053) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.203462 s, 82.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210763 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348502 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0212499 s, 49.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0287919 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0601011 s, 17.4 MB/s , 0.0718377 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00809817 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00837146 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0590613 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113111 s, 92.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0735253 s, 14.3 MB/s , 0.0762606 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112174 s, 93.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0650841 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011866 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0115276 s, 91.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117621 s, 89.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138728 s, 75.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122613 s, 85.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147651 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137995 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138804 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148714 s, 70.5 MB/s , 0.0152705 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136898 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0162207 s, 64.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0728774 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0775818 s, 13.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.115246 s, 9.1 MB/s , 0.124327 s, 8.4 MB/s , 0.114999 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.140165 s, 7.5 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.143962 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106774 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0270339 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0175231 s, 59.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0319843 s, 32.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0388729 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0637392 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.073476 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0656088 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0771643 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0875267 s, 12.0 MB/s , 0.0837451 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117977 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103099 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126766 s, 82.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108702 s, 96.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00362921 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133021 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117509 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110417 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111407 s, 94.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100367 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117726 s, 89.1 MB/s , 0.00833217 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131998 s, 79.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0119231 s, 87.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0127201 s, 82.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113747 s, 92.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100948 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.118716 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.117169 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.140957 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134026 s, 7.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.169474 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00728741 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215621 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0375692 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00943973 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0111644 s, 93.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0086573 s, 121 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00643097 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108351 s, 96.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073003 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105614 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0755547 s, 13.9 MB/s , 0.0832802 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00852879 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0816969 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106025 s, 98.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00895025 s, 117 MB/s , 0.0118528 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0908418 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109681 s, 95.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0899134 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117494 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148364 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135851 s, 77.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151166 s, 69.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122939 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123805 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.124677 s, 8.4 MB/s , 0.125734 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136441 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.131901 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.154248 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.144441 s, 7.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (3s) == sanityn test 77d: check TRR nrs policy ================ 13:17:36 (1731349056) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.106017 s, 158 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0074118 s, 141 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208596 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148092 s, 70.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0262398 s, 40.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0305234 s, 34.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0589685 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00949311 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00792857 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00815385 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00931286 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00963254 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0683243 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00991242 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111671 s, 93.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106067 s, 98.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.011865 s, 88.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138177 s, 75.9 MB/s , 0.012515 s, 83.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128784 s, 81.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0129219 s, 81.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0138809 s, 75.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126379 s, 83.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0146823 s, 71.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0964119 s, 10.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0961548 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.102998 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.107835 s, 9.7 MB/s 1048576 bytes (1.0 MB) copied, 0.103826 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.106878 s, 9.8 MB/s , 0.111906 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.129068 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13337 s, 7.9 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.141892 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125048 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0125838 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.063518 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0494944 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0594736 s, 17.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0569296 s, 18.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0881053 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0915421 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0947178 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199871 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0185678 s, 56.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196254 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0187046 s, 56.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196398 s, 53.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0188909 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.019085 s, 54.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211995 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0224434 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0211859 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208908 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.021733 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0223852 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0215308 s, 48.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0234239 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014762 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106608 s, 9.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.115373 s, 9.1 MB/s , 0.121316 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.142785 s, 7.3 MB/s 1+0 records in 1+0 records out , 0.147631 s, 7.1 MB/s 1048576 bytes (1.0 MB) copied, 0.15039 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153728 s, 6.8 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.152471 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150901 s, 69.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0368354 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.038047 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331419 s, 31.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420512 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0517191 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0482497 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107364 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110663 s, 94.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124989 s, 83.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0123931 s, 84.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121705 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121167 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127806 s, 82.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00449102 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0018514 s, 566 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0875157 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0207152 s, 50.6 MB/s , 0.00668919 s, 157 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00168126 s, 624 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217925 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838784 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196986 s, 53.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0182722 s, 57.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0230862 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.091376 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0914543 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0961323 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.106064 s, 9.9 MB/s , 0.118865 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135375 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.139723 s, 7.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (4s) == sanityn test 77e: check TBF NID nrs policy ============ 13:17:40 (1731349060) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.32@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0737125 s, 228 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0339765 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370417 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0296088 s, 35.4 MB/s 1048576 bytes (1.0 MB) copied, 0.031037 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0403843 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0370526 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10467 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115635 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.140765 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159065 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179681 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136268 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.133661 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.133146 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134244 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135065 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134743 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134569 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.137294 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.13911 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.135735 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.1366 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136328 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.1369 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136183 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.137572 s, 7.6 MB/s , 0.138377 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.216661 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.245756 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.269614 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.284274 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.30989 s, 3.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.190631 s, 88.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00786804 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135481 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0218086 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0303898 s, 34.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0293003 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0340326 s, 30.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.057794 s, 18.1 MB/s , 0.0583472 s, 18.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0627537 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0588651 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0519542 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0738029 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.136884 s, 7.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.165975 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.167064 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.197686 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0643355 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0621727 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.061858 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0625512 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0642648 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0652495 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0650378 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0635642 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0631142 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0637618 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0642909 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0642901 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0641514 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0641931 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0641309 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.06684 s, 15.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.123882 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0109997 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965393 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.051272 s, 20.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0590245 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0643181 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611696 s, 17.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00911821 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.00950265 s, 110 MB/s 1048576 bytes (1.0 MB) copied, 0.00965842 s, 109 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130074 s, 80.6 MB/s , 0.00959583 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112341 s, 93.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114127 s, 91.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118992 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134817 s, 77.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012557 s, 83.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0178525 s, 58.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0131373 s, 79.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0915766 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0885907 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0136523 s, 76.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0900122 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150608 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0158193 s, 66.3 MB/s , 0.0131128 s, 80.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104628 s, 10.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.118564 s, 8.8 MB/s , 0.115167 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123648 s, 8.5 MB/s , 0.118327 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.123529 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122877 s, 8.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.111462 s, 151 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00718897 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118343 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0210251 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0207261 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0359166 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0511924 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0455494 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0496969 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0536305 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0534854 s, 19.6 MB/s 1048576 bytes (1.0 MB) copied, 0.0334157 s, 31.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00537448 s, 195 MB/s , 0.00628284 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119614 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00939821 s, 112 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0074041 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0108406 s, 96.7 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0756917 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00928716 s, 113 MB/s 1048576 bytes (1.0 MB) copied, 0.00844965 s, 124 MB/s 1+0 records in 1+0 records out , 0.00811043 s, 129 MB/s 1048576 bytes (1.0 MB) copied, 0.00980899 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0105885 s, 99.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100194 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00965521 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107277 s, 97.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110075 s, 95.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107519 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0900557 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838228 s, 12.5 MB/s , 0.0648026 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0836857 s, 12.5 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77e.sanityn' PASS 77e (5s) == sanityn test 77f: check TBF JobID nrs policy ========== 13:17:45 (1731349065) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.302239 s, 55.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252267 s, 41.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0231254 s, 45.3 MB/s , 0.0232113 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0331195 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0462518 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0391944 s, 26.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0454495 s, 23.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00769325 s, 136 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00141065 s, 743 MB/s , 0.00877198 s, 120 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111761 s, 93.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0822418 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838225 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0854915 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00782786 s, 134 MB/s , 0.00907703 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00764652 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0899272 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.091717 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0169898 s, 61.7 MB/s , 0.0248958 s, 42.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00186023 s, 564 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0101533 s, 103 MB/s , 0.00523999 s, 200 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00617606 s, 170 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00173271 s, 605 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00461422 s, 227 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00449277 s, 233 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110642 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.127743 s, 8.2 MB/s , 0.119564 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.121831 s, 8.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.208852 s, 80.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124356 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0266992 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.030349 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611395 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0861476 s, 12.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00717015 s, 146 MB/s , 0.0077929 s, 135 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073412 s, 143 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102337 s, 102 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00758859 s, 138 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00833599 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00127679 s, 821 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102425 s, 102 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00983198 s, 107 MB/s , 0.00999268 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00979093 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102045 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0102247 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106143 s, 98.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0107579 s, 97.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106077 s, 98.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.152198 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.155974 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159876 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.155873 s, 6.7 MB/s 1048576 bytes (1.0 MB) copied, 0.157605 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159957 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.163524 s, 6.4 MB/s 1048576 bytes (1.0 MB) copied, 0.164016 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.178446 s, 5.9 MB/s , 0.184784 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.182821 s, 5.7 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.0824635 s, 203 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0068916 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00851184 s, 123 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0219675 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0240379 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263504 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0296921 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.027329 s, 38.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.03284 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0611049 s, 17.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0568978 s, 18.4 MB/s , 0.0541169 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0691929 s, 15.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0772618 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0708879 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0896353 s, 11.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00695754 s, 151 MB/s 1048576 bytes (1.0 MB) copied, 0.00787024 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00737886 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00963763 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00804957 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00795826 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00184495 s, 568 MB/s 1048576 bytes (1.0 MB) copied, 0.0118538 s, 88.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00961289 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111233 s, 94.3 MB/s 1048576 bytes (1.0 MB) copied, 0.0111234 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119778 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0920411 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0112753 s, 93.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132918 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127007 s, 82.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.012971 s, 80.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.127631 s, 131 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0384838 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0404985 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0422163 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0408675 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0494568 s, 21.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0546038 s, 19.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00589644 s, 178 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00901757 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0017685 s, 593 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104376 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0140754 s, 74.5 MB/s , 0.0130226 s, 80.5 MB/s , 0.0218395 s, 48.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00182459 s, 575 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0162399 s, 64.6 MB/s , 0.0201071 s, 52.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00839715 s, 125 MB/s , 0.0073901 s, 142 MB/s , 0.00659452 s, 159 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00290554 s, 361 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00669022 s, 157 MB/s , 0.00573475 s, 183 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115288 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.158085 s, 6.6 MB/s , 0.159296 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.150824 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.153015 s, 6.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.179709 s, 5.8 MB/s , 0.173546 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.171697 s, 6.1 MB/s 1048576 bytes (1.0 MB) copied, 0.186427 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.179226 s, 5.9 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77f.sanityn' PASS 77f (6s) == sanityn test 77g: Change TBF type directly ============ 13:17:51 (1731349071) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (3s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 13:17:54 (1731349074) oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Unknown error 524 oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Unknown error 524 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 12 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Unknown error 524 oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Unknown error 524 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 12 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Unknown error 524 oleg232-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Unknown error 524 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 12 PASS 77h (3s) == sanityn test 77i: Change rank of TBF rule ============= 13:17:57 (1731349077) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (6s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 13:18:03 (1731349083) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.596702 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134937 s, 77.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0263262 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0885744 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.1693 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.214353 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.287981 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.324584 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.187166 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.189103 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.186683 s, 5.6 MB/s 1048576 bytes (1.0 MB) copied, 0.187619 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.189414 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.188904 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.19339 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.19045 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.19103 s, 5.5 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.191632 s, 5.5 MB/s , 0.192351 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.192834 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.19231 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.192951 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.192847 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.194987 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.368962 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.42221 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.471865 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.527846 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.565574 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.624906 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.663448 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.71915 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.773566 s, 1.4 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.90482 s, 21.4 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.436 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (50s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 13:18:53 (1731349133) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 2.64391 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0316049 s, 33.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0421289 s, 24.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0536069 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0764224 s, 13.7 MB/s , 0.0678325 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0681338 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.064245 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0676737 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.374018 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.378028 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.385872 s, 2.7 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.406565 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.41621 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00785564 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0069335 s, 151 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.403575 s, 2.6 MB/s 1048576 bytes (1.0 MB) copied, 0.428506 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121415 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00987844 s, 106 MB/s 1048576 bytes (1.0 MB) copied, 0.0100402 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.427991 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0113987 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118406 s, 88.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0121193 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0184102 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0130009 s, 80.7 MB/s 1048576 bytes (1.0 MB) copied, 0.019077 s, 55.0 MB/s 1048576 bytes (1.0 MB) copied, 0.0158321 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.014641 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0149056 s, 70.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0147269 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0145148 s, 72.2 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8294 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9672 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 2.6385 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0279573 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0420501 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.085123 s, 12.3 MB/s , 0.0842496 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100738 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0814267 s, 12.9 MB/s 1048576 bytes (1.0 MB) copied, 0.092595 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100876 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.377947 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.377917 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0101943 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.010586 s, 99.1 MB/s , 0.0109337 s, 95.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0114345 s, 91.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0120198 s, 87.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0122714 s, 85.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0118894 s, 88.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.372505 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.401599 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.404011 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146354 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0137186 s, 76.4 MB/s 1048576 bytes (1.0 MB) copied, 0.0148508 s, 70.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140003 s, 74.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0135593 s, 77.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0153695 s, 68.2 MB/s , 0.0153812 s, 68.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0153064 s, 68.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0182891 s, 57.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.389399 s, 2.7 MB/s , 0.406175 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.390723 s, 2.7 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 20.0254 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9627 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (131s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 13:21:04 (1731349264) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 2.64056 s, 6.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0312765 s, 33.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0509657 s, 20.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0897442 s, 11.7 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.097217 s, 10.8 MB/s , 0.0910559 s, 11.5 MB/s 1048576 bytes (1.0 MB) copied, 0.0880499 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0873556 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0827363 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.372622 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.373452 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.364226 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.384971 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.385027 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.37111 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.384692 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.380304 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00817174 s, 128 MB/s , 0.00730675 s, 144 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00847683 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00901582 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00848775 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00811682 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00832391 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00134574 s, 779 MB/s 1048576 bytes (1.0 MB) copied, 0.00803917 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00914534 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00866615 s, 121 MB/s 1048576 bytes (1.0 MB) copied, 0.00835308 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00903386 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00920378 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00891706 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100727 s, 104 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 20.0451 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9281 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 2.64735 s, 6.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0549319 s, 19.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.102061 s, 10.3 MB/s , 0.0731799 s, 14.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.121345 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.132479 s, 7.9 MB/s , 0.133248 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.11989 s, 8.7 MB/s , 0.123066 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.401336 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.423141 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.423734 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.404851 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.426382 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.422668 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.41551 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.413869 s, 2.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0155506 s, 67.4 MB/s , 0.0157542 s, 66.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173519 s, 60.4 MB/s 1048576 bytes (1.0 MB) copied, 0.018199 s, 57.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0254202 s, 41.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0193777 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0252532 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199062 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0204391 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0248011 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0206645 s, 50.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0208445 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0237919 s, 44.1 MB/s 1+0 records in 1+0 records out , 0.0219357 s, 47.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0220037 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0230485 s, 45.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.8342 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.9695 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (130s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 13:23:14 (1731349394) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.634102 s, 26.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0522911 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0415709 s, 25.2 MB/s 1048576 bytes (1.0 MB) copied, 0.0416096 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0537748 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0431453 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0348651 s, 30.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0746452 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103862 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.101885 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0891807 s, 11.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.00776102 s, 135 MB/s , 0.00531606 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.100796 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.116141 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0186574 s, 56.2 MB/s , 0.024869 s, 42.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00169926 s, 617 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0223735 s, 46.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0183578 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00199381 s, 526 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0284587 s, 36.8 MB/s 1048576 bytes (1.0 MB) copied, 0.0226183 s, 46.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0177511 s, 59.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0116018 s, 90.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00902861 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10038 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00818442 s, 128 MB/s 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.131477 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0935695 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00828606 s, 127 MB/s , 0.00916747 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.125607 s, 8.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.92646 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75154 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.32@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.32@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.707317 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0199007 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0217642 s, 48.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0734703 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0548002 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0708713 s, 14.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11613 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122527 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.128158 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111824 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110738 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.108931 s, 9.6 MB/s , 0.10874 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111243 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.113706 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11449 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.11491 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.116129 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119268 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.119731 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.120339 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.120506 s, 8.7 MB/s , 0.120966 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122224 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.124008 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.351885 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.410937 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.456018 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.514754 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.589034 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.639115 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.679978 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.729154 s, 1.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95456 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74012 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.32@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.648158 s, 25.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0585893 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0565022 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0695632 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0837315 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0793927 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0804334 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0200699 s, 52.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out , 0.0169893 s, 61.7 MB/s 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0185392 s, 56.6 MB/s , 0.0117541 s, 89.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0157501 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140541 s, 74.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0196737 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.12567 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0213615 s, 49.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0171343 s, 61.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0173252 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159263 s, 65.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.013297 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143723 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0146245 s, 71.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0148897 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0159827 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.150797 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.169295 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.159737 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.176259 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.195853 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.21398 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.218712 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.201883 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.218475 s, 4.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95837 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.8909 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.821975 s, 20.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0134308 s, 78.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.018258 s, 57.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.134594 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.183114 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.222082 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.265728 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.320701 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.372209 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.405083 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.457551 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.558301 s, 1.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.559973 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.625036 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.665471 s, 1.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10734 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106946 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106821 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.112773 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107242 s, 9.8 MB/s 1048576 bytes (1.0 MB) copied, 0.111977 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108791 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10852 s, 9.7 MB/s , 0.106974 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110162 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108673 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109015 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.110574 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111907 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111893 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115238 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.728408 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.772604 s, 1.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.84626 s, 10.6 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.75235 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95028 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74712 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 2.31477 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0507215 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.110797 s, 9.5 MB/s 1048576 bytes (1.0 MB) copied, 0.109058 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.15543 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.156418 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.188214 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.184859 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.205147 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.360873 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.366052 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.357408 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=2 running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.402684 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0125606 s, 83.5 MB/s 1048576 bytes (1.0 MB) copied, 0.00622626 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00688425 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0073418 s, 143 MB/s 1048576 bytes (1.0 MB) copied, 0.0159693 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100872 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00193012 s, 543 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0103655 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00211763 s, 495 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0100522 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00217599 s, 482 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124802 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0124044 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.435125 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.415978 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.417031 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00305575 s, 343 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0017904 s, 586 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0150547 s, 69.7 MB/s , 0.437348 s, 2.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.681 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 19.4451 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.604926 s, 27.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0304223 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0565036 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.056154 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0727158 s, 14.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0855597 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0782534 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0849479 s, 12.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0953029 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00690803 s, 152 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00901876 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.00853501 s, 123 MB/s 1048576 bytes (1.0 MB) copied, 0.00866358 s, 121 MB/s 1048576 bytes (1.0 MB) copied, 0.0088892 s, 118 MB/s 1048576 bytes (1.0 MB) copied, 0.0121262 s, 86.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0119066 s, 88.1 MB/s 1048576 bytes (1.0 MB) copied, 0.0135495 s, 77.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0104199 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.118927 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114407 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111621 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.117056 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111023 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111072 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0126339 s, 83.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0133473 s, 78.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0140284 s, 74.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0142559 s, 73.6 MB/s , 0.0143796 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151676 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015027 s, 69.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.130048 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.122704 s, 8.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.94866 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74564 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 1.19111 s, 14.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_w] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0111192 s, 94.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0910932 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108933 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.162753 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.19794 s, 5.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.210241 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.207119 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.206681 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.20857 s, 5.0 MB/s 1048576 bytes (1.0 MB) copied, 0.207567 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.211427 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.221004 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.223307 s, 4.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: [declare] [-a] [pids_r] execvp fails running declare (2): No such file or directory 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.235086 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.289145 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104417 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.103842 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104264 s, 10.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104367 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.104862 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107604 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10427 s, 10.1 MB/s 1048576 bytes (1.0 MB) copied, 0.1051 s, 10.0 MB/s 1048576 bytes (1.0 MB) copied, 0.106477 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106516 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106579 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106571 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.106951 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108213 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.108171 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.109801 s, 9.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88012aaf6000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.612005 s, 1.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.80291 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74347 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.7714 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 9.74301 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (352s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 13:29:06 (1731349746) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (4s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 13:29:10 (1731349750) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg232-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 3.64455 s, 288 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg232-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 40.0326 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (50s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 13:30:00 (1731349800) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.228193 s, 73.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00749245 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0189803 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0370653 s, 28.3 MB/s , 0.0335689 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0228836 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0492613 s, 21.3 MB/s , 0.048566 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0501588 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00755123 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0596825 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00643034 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.063208 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0117101 s, 89.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.00984184 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1048576 bytes (1.0 MB) copied, 0.0107048 s, 98.0 MB/s , 0.0100227 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0106785 s, 98.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0110245 s, 95.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0127701 s, 82.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0128599 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0132866 s, 78.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88012aaf6000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0143677 s, 73.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0152274 s, 68.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0151839 s, 69.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.015734 s, 66.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0164009 s, 63.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out , 0.0813185 s, 12.9 MB/s 1048576 bytes (1.0 MB) copied, 0.0730764 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.091743 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0889738 s, 11.8 MB/s , 0.11308 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.111654 s, 9.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg232-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.98169 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97495 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB) copied, 0.707053 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0770686 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0806567 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.074964 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.10435 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0838835 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0976025 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0959495 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.114869 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.115671 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.107454 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.24865 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.251044 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.253188 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.251631 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.255975 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.251119 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.254219 s, 4.1 MB/s , 0.255252 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.254842 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.255718 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.255646 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.256457 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.257036 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.258462 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.257963 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.263038 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.553702 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.603613 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.656323 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.70252 s, 1.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.752868 s, 1.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.812359 s, 1.3 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg232-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg232-client.virtnet' removed directory: '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.95481 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB) copied, 4.97426 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (64s) == sanityn test 77o: Changing rank should not panic ====== 13:31:04 (1731349864) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (5s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 13:31:09 (1731349869) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (35s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 13:31:44 (1731349904) mds.MDS.mdt.nrs_policies=tbf oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg232-client: oleg232-server: ssh exited with exit code 36 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg232-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg232-client: oleg232-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 13:31:57 (1731349917) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 9234 kill md_thread xargs PASS 77r (38s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 13:32:35 (1731349955) oleg232-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg232-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg232-client: oleg232-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (4s) == sanityn test 79: xattr: intent error ================== 13:32:39 (1731349959) File: '/mnt/lustre/d79.sanityn' Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506861 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:32:39.000000000 -0500 Modify: 2024-11-11 13:32:39.000000000 -0500 Change: 2024-11-11 13:32:39.000000000 -0500 Birth: - fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (12s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 13:32:52 (1731349972) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 13:32:53 (1731349973) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 13:32:54 (1731349974) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 13:32:55 (1731349975) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 13:32:56 (1731349976) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 13:32:57 (1731349977) total: 2500 open/close in 9.34 seconds: 267.62 ops/second snapshot_time 1731350007.719576377 secs.nsecs start_time 1731349978.985310216 secs.nsecs elapsed_time 28.734266161 secs.nsecs open 2505 samples [usecs] 95 1804 1141263 552145443 close 2504 samples [usecs] 51 2025 347802 55094740 mknod 2501 samples [usecs] 154 1649 1017810 440981886 unlink 2500 samples [usecs] 268 16844 1835757 3527778215 mkdir 2 samples [usecs] 740 1067 1807 1686089 rmdir 2 samples [usecs] 2383 4955 7338 30230714 rename 2500 samples [usecs] 220 5024 2721934 5253081626 getattr 3761 samples [usecs] 8 612 81098 2454850 parallel_rename_file 2500 samples [usecs] 220 5024 2721934 5253081626 crossdir_rename 2500 samples [usecs] 220 5024 2721934 5253081626 PASS 81d (32s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 13:33:29 (1731350009) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.6953 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.6953 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 13:33:31 (1731350011) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 13:33:32 (1731350012) fail_loc=0x8000060b PASS 84 (7s) == sanityn test 85: Lustre API root cache race =========== 13:33:39 (1731350019) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1731350020 Finishing test test1 at 1731350020 PASS 85 (3s) == sanityn test 90: open/create and unlink striped directory ========================================================== 13:33:42 (1731350022) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (1s) == sanityn test 91: chmod and unlink striped directory === 13:33:43 (1731350023) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 13:33:44 (1731350024) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 13:33:45 (1731350025) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6256 0x1870 0x240000400 1 6095 0x17cf 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6257 0x1871 0x240000400 1 6096 0x17d0 0x280000400 rc1=0 and rc2=0 PASS 93 (9s) == sanityn test 94: signal vs CP callback race =========== 13:33:54 (1731350034) osc.lustre-OST0000-osc-ffff8800a9dfc800.idle_timeout=0 osc.lustre-OST0000-osc-ffff88012aaf6000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8800a9dfc800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88012aaf6000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.0466477 s, 176 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 3.0194 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5743: 30849 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0238805 s, 172 kB/s osc.lustre-OST0000-osc-ffff8800a9dfc800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012aaf6000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9dfc800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aaf6000.idle_timeout=debug PASS 94 (10s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 13:34:04 (1731350044) llite.lustre-ffff8800a9dfc800.fast_read=0 llite.lustre-ffff88012aaf6000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.0545014 s, 38.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8800a9dfc800.fast_read=1 llite.lustre-ffff88012aaf6000.fast_read=1 PASS 95a (19s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 13:34:24 (1731350064) llite.lustre-ffff8800a9dfc800.fast_read=0 llite.lustre-ffff88012aaf6000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB) copied, 0.00208046 s, 5.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8800a9dfc800.fast_read=1 llite.lustre-ffff88012aaf6000.fast_read=1 PASS 95b (9s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 13:34:32 (1731350072) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 13:34:33 (1731350073) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00257579 s, 1.6 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (3s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 13:34:36 (1731350076) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0312207 s, 67.2 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 13:34:38 (1731350078) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB) copied, 0.0577453 s, 36.3 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (3s) == sanityn test 100e: DoM: read on open and file size ==== 13:34:41 (1731350081) 1+0 records in 1+0 records out 12000 bytes (12 kB) copied, 0.00315522 s, 3.8 MB/s PASS 100e (2s) == sanityn test 101a: Discard DoM data on unlink ========= 13:34:43 (1731350083) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00102667 s, 4.0 MB/s PASS 101a (3s) == sanityn test 101b: Discard DoM data on rename ========= 13:34:46 (1731350086) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.000545682 s, 7.5 MB/s PASS 101b (2s) == sanityn test 101c: Discard DoM data on close-unlink === 13:34:48 (1731350088) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00070365 s, 5.8 MB/s PASS 101c (4s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 13:34:52 (1731350092) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1731350093 Finishing test test23 at 1731350098 This doesn't work 100%, but this is just reproducing the bug, not testing the fix, so OK to not fail test. fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1731350100 Finishing test test23 at 1731350103 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 13:35:06 (1731350106) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (2s) == sanityn test 105: Glimpse and lock cancel race ======== 13:35:08 (1731350108) fail_loc=0x416 PASS 105 (22s) == sanityn test 106a: Verify the btime via statx() ======= 13:35:30 (1731350130) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 13:35:31 (1731350131) SKIP: sanityn test_106b statx() only test SKIP 106b (1s) == sanityn test 106c: Verify statx attributes mask ======= 13:35:32 (1731350132) SKIP: sanityn test_106c statx() only test SKIP 106c (1s) == sanityn test 107a: Basic grouplock conflict =========== 13:35:34 (1731350134) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.314808 s, 33.3 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6101 0x17d5 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 First grouplock blocks second one PASS 107a (4s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 13:35:38 (1731350138) 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.375018 s, 28.0 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6265 0x1879 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.6953 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (9s) == sanityn test 108a: lseek: parallel updates ============ 13:35:47 (1731350147) 1+0 records in 1+0 records out 10000 bytes (10 kB) copied, 0.0106944 s, 935 kB/s fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB) copied, 22.6505 s, 370 kB/s PASS 108a (24s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 13:36:11 (1731350171) 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 4 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Iteration 5 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 6 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 19 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 20 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 23 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 30 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 31 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 33 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Iteration 36 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Iteration 39 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 192.168.202.132@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 192.168.202.132@tcp:/lustre /mnt/lustre2 lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project 0 0 Stopping client oleg232-client.virtnet /mnt/lustre (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 (opts:) Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre2 PASS 109 (416s) == sanityn test 110: do not grant another lock on resend ========================================================== 13:43:07 (1731350587) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: '/mnt/lustre/f110.sanityn' Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-11 13:43:07.000000000 -0500 Modify: 2024-11-11 13:43:07.000000000 -0500 Change: 2024-11-11 13:43:07.000000000 -0500 Birth: - fail_loc=0 fail_val=0 Connected clients: oleg232-client.virtnet oleg232-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 13:44:12 (1731350652) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 13:44:13 (1731350653) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 13:44:14 (1731350654) lustre-OST0000-osc-ffff8800b6f48800 active. lustre-OST0001-osc-ffff8800b6f48800 active. lustre-OST0000-osc-ffff8800b6f48800 active. lustre-OST0001-osc-ffff8800b6f48800 active. MGC192.168.202.132@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 13:44:16 (1731350656) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 13:44:17 (1731350657) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 13:44:18 (1731350658) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 13:44:19 (1731350659) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=24 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.014 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.0187 s, 0.2 kB/s , 20.023 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 20.125 s, 0.2 kB/s , 20.1131 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 20.1134 s, 0.2 kB/s , 20.1187 s, 0.2 kB/s , 20.1161 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 20.1194 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.0189 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1256 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1302 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1611 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1817 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1876 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 40.1659 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 40.2157 s, 0.1 kB/s , 40.1988 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.1265 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2051 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2264 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2225 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.3134 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2564 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 60.2965 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 60.3278 s, 0.1 kB/s 1+0 records in 1+0 records out , 60.2613 s, 0.1 kB/s 4096 bytes (4.1 kB) copied, 60.3031 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.2965 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.2877 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3172 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3642 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.4103 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3795 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.383 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.4154 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 80.3728 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 100.433 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 100.458 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 100.459 s, 0.0 kB/s , 100.467 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 100.476 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied1+0 records in 1+0 records out , 100.479 s, 0.0 kB/s 4096 bytes (4.1 kB) copied, 100.517 s, 0.0 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied4096 bytes (4.1 kB) copied, 100.5 s, 0.0 kB/s , 100.511 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 120.475 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 120.488 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg232-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8800b3824000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800b3824000.ost_server_uuid in IDLE state after 0 sec PASS 200 (127s) cleanup: ====================================================== == sanityn test complete, duration 4705 sec ============== 13:46:26 (1731350786) sanityn: FAIL: test_16a fsx failed sanityn: FAIL: test_16b fsx with tiny write failed. sanityn: FAIL: test_16k - fsx 2 FAILURE! rc=1 sanityn: FAIL: test_18 mmap_sanity test failed === sanityn: start cleanup 13:46:27 (1731350787) === Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg232-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 13:47:20 (1731350840) ===