-----============= acceptance-small: sanityn ============----- Thu Oct 23 23:35:38 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg445-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 23:36:02 (1761276962) === Starting client oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Mount client oleg445-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Started clients oleg445-client.virtnet: 192.168.204.145@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg445-client.virtnet: executing check_config_client /mnt/lustre oleg445-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg445-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff99e1885c6800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99e1a10ae000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99e1885c6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99e1a10ae000.idle_timeout=debug disable quota as required oleg445-server: oleg445-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 23:36:36 (1761276996) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300602 s, 34.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7551] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 23:36:39 (1761276999) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 23:36:47 (1761277007) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (5s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 23:36:52 (1761277012) -rw-r--r-- 1 root root 0 Oct 23 23:36 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 23:36:58 (1761277018) -rw-r--r-- 1 root root 0 Oct 23 23:37 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 23:37:05 (1761277025) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (7s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 23:37:12 (1761277032) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (7s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 23:37:19 (1761277039) total 1 -rwxrwxrwx 1 root root 0 Oct 23 23:36 f2a -rwxrwxrwx 1 root root 0 Oct 23 23:37 f2b total 1 -rwxrwxrwx 1 root root 0 Oct 23 23:36 f2a -rwxrwxrwx 1 root root 0 Oct 23 23:37 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0139754 s, 36.6 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 23:37:26 (1761277046) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 23:37:30 (1761277050) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.337334 s, 6.2 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (6s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 23:37:36 (1761277056) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 23:37:42 (1761277062) PASS 4 (6s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 23:37:48 (1761277068) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (5s) == sanityn test 6: remove of open file on other node ============================================================================ 23:37:53 (1761277073) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (5s) == sanityn test 7: remove of open directory on other node ======================================================================= 23:37:58 (1761277078) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 23:38:03 (1761277083) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (4s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 23:38:07 (1761277087) PASS 9a (5s) == sanityn test 9b: append to striped sparse file ======== 23:38:12 (1761277092) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022207 s, 47.2 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0351262 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (4s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 23:38:16 (1761277096) 1+0 records in 1+0 records out 1 byte copied, 0.00778866 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00741829 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00856618 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00674198 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0243861 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0111787 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103479 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114202 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00682412 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00698809 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00690821 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00687867 s, 0.1 kB/s PASS 10a (4s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 23:38:20 (1761277100) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00491321 s, 625 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0168092 s, 244 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000204558 s, 15.0 MB/s PASS 10b (3s) == sanityn test 11: execution of file opened for write should return error ============================================================== 23:38:23 (1761277103) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 23:38:26 (1761277106) warning: '-runtime' deprecated, use '-t runtime' instead - link 1482 (time 1761277116.84 total 10.00 last 148.14) - link 3086 (time 1761277126.84 total 20.01 last 160.38) - link 4555 (time 1761277136.85 total 30.01 last 146.82) - link 6040 (time 1761277146.85 total 40.02 last 148.42) - link 7967 (time 1761277156.86 total 50.02 last 192.63) - link 9674 (time 1761277166.86 total 60.02 last 170.63) - link 10000 (time 1761277169.81 total 62.97 last 110.54) - link 10702 (time 1761277179.81 total 72.98 last 70.18) - link 11416 (time 1761277189.82 total 82.98 last 71.37) - link 13182 (time 1761277199.82 total 92.98 last 176.60) total: 14543 link in 100.01 seconds: 145.42 ops/second using seed 3654548246 running for 100 seconds - stat 10000 (time 1761277116 ; total 9 ; last 9) - stat 20000 (time 1761277124 ; total 17 ; last 8) - stat 30000 (time 1761277133 ; total 26 ; last 9) - stat 40000 (time 1761277141 ; total 34 ; last 8) - stat 50000 (time 1761277150 ; total 43 ; last 9) - stat 60000 (time 1761277158 ; total 51 ; last 8) - stat 65451 (time 1761277169 ; total 62 ; last 11) - stat 70000 (time 1761277177 ; total 70 ; last 8) - stat 74937 (time 1761277188 ; total 81 ; last 11) - stat 80000 (time 1761277198 ; total 91 ; last 10) total: 84485 stats in 100 seconds: 844.849976 stats/second - unlinked 0 (time 1761277157 ; total 0 ; last 0) - unlinked 10000 (time 1761277226 ; total 69 ; last 69) unlink(/mnt/lustre2/lockdir/lockfile14543) error: No such file or directory total: 14542 unlinks in 85 seconds: 171.082352 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21347) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21349) - No such process PASS 12 (138s) == sanityn test 13: test directory page revocation ======= 23:40:44 (1761277244) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 23:40:47 (1761277247) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 22533 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 23:40:49 (1761277249) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23111 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 23:40:51 (1761277251) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23689 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 23:40:53 (1761277253) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24270 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 23:40:55 (1761277255) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24851 Terminated $DIR1/$tdir/sleep 60 PASS 14d (3s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 23:40:58 (1761277258) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7524352 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7524352KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (29s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 23:41:27 (1761277287) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0864194 s, 97.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0669455 s, 125 MB/s Chance of close/open is 1 in 50 Seed set to 7288 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1761277288.502340 mapwrite 0x52331a thru 0x528210 (0x4ef7 bytes) 000003[1] 1761277288.531869 write 0x2f6e38 thru 0x300130 (0x92f9 bytes) 000004[1] 1761277288.536069 read 0x0dc174 thru 0x0e0e81 (0x4d0e bytes) 000005[1] 1761277288.543790 mapread 0x35547e thru 0x35bdd2 (0x6955 bytes) 000007[0] 1761277288.552357 read 0x0ae270 thru 0x0b6874 (0x8605 bytes) 000010[0] 1761277288.560583 read 0x12e5d9 thru 0x12ff28 (0x1950 bytes) 000011[0] 1761277288.565791 mapread 0x4e6fb0 thru 0x4f4edb (0xdf2c bytes) 000013[0] 1761277288.579019 read 0x39f01f thru 0x3abcda (0xccbc bytes) 000015[1] 1761277288.582368 mapwrite 0x5b243f thru 0x5c2168 (0xfd2a bytes) truncating to largest ever: 0x3b437d 000017[0] 1761277288.622983 trunc from 0x5c2169 to 0x3b437c (0x20ddec bytes) 000022[0] 1761277288.638578 read 0x13a2d2 thru 0x149c09 (0xf938 bytes) 000023[1] 1761277288.665470 read 0x248701 thru 0x24f687 (0x6f87 bytes) 000024[1] 1761277288.697465 write 0x74c40c thru 0x7544b4 (0x80a9 bytes) 000026[0] 1761277288.702569 mapwrite 0x815421 thru 0x818bc7 (0x37a7 bytes) truncating to largest ever: 0x906f84 000029[1] 1761277288.780736 trunc from 0x818bc8 to 0x906f83 (0xee3bc bytes) 000030[1] 1761277288.794554 mapwrite 0x9d3f80 thru 0x9deb35 (0xabb6 bytes) 000032[1] 1761277288.874756 mapread 0x73358d thru 0x739516 (0x5f8a bytes) 000033[1] 1761277288.878959 mapread 0x2de5f0 thru 0x2e95eb (0xaffc bytes) 000034[0] 1761277288.894913 write 0x48a3da thru 0x495f28 (0xbb4f bytes) 000036[0] 1761277288.911345 write 0x693369 thru 0x69c5ba (0x9252 bytes) 000038[1] 1761277288.922046 trunc from 0x9deb36 to 0x7517ba (0x28d37b bytes) 000039[1] 1761277288.932971 mapread 0x30271f thru 0x30a0db (0x79bd bytes) 000041[1] 1761277288.948188 write 0x55b5d4 thru 0x55c00c (0xa39 bytes) 000043[1] 1761277288.954918 trunc from 0x7517bb to 0x7b3ad9 (0x6231f bytes) 000045[0] 1761277288.964934 write 0x42f207 thru 0x438cb8 (0x9ab2 bytes) 000050[0] 1761277288.987529 mapread 0x4f8db8 thru 0x506630 (0xd879 bytes) 000053[1] 1761277289.015585 mapwrite 0x446977 thru 0x44d226 (0x68b0 bytes) 000056[0] 1761277289.054028 read 0x3d3518 thru 0x3d3eb9 (0x9a2 bytes) 000057[1] 1761277289.059923 write 0x0a3061 thru 0x0b12d8 (0xe278 bytes) 000058[1] 1761277289.065816 mapread 0x490fcc thru 0x496e19 (0x5e4e bytes) 000059[0] 1761277289.077863 mapread 0x698f29 thru 0x6a789f (0xe977 bytes) 000061[1] 1761277289.092928 read 0x7115ab thru 0x719cdd (0x8733 bytes) 000062[1] 1761277289.095293 mapread 0x0bd123 thru 0x0c68da (0x97b8 bytes) 000063[1] 1761277289.103318 write 0x6a2639 thru 0x6a7284 (0x4c4c bytes) 000064[1] 1761277289.108548 write 0x241f9b thru 0x24761d (0x5683 bytes) 000065[0] 1761277289.111097 mapread 0x49f27c thru 0x4a9592 (0xa317 bytes) 000067[0] 1761277289.129672 write 0x32cd13 thru 0x332326 (0x5614 bytes) 000069[0] 1761277289.141105 mapread 0x17ed2a thru 0x180294 (0x156b bytes) 000072[1] 1761277289.150977 write 0x2c05c4 thru 0x2cc808 (0xc245 bytes) 000073[0] 1761277289.166335 mapwrite 0x5d636b thru 0x5d80ba (0x1d50 bytes) 000074[0] 1761277289.190119 trunc from 0x7b3ada to 0x751ec9 (0x61c10 bytes) 000076[1] 1761277289.206000 trunc from 0x751eca to 0x86184c (0x10f983 bytes) 000077[1] 1761277289.216103 mapwrite 0x5e8d22 thru 0x5f393c (0xac1b bytes) 000084[0] 1761277289.248886 trunc from 0x86184d to 0x62fe95 (0x2319b7 bytes) 000085[1] 1761277289.259781 write 0x097023 thru 0x0a2032 (0xb010 bytes) 000086[1] 1761277289.269737 mapwrite 0x494f26 thru 0x49a609 (0x56e4 bytes) 000088[0] 1761277289.301517 read 0x45372c thru 0x45fe1e (0xc6f3 bytes) 000089[1] 1761277289.308536 read 0x08e3bf thru 0x0938bc (0x54fe bytes) 000093[1] 1761277289.322539 mapwrite 0x0308bb thru 0x0378e1 (0x7027 bytes) 000094[1] 1761277289.351791 write 0x340125 thru 0x3405d5 (0x4b1 bytes) 000096[1] 1761277289.361230 write 0x28a641 thru 0x295c4d (0xb60d bytes) truncating to largest ever: 0x99e6b5 000097[1] 1761277289.365013 trunc from 0x62fe96 to 0x99e6b4 (0x36e81f bytes) 000098[0] 1761277289.374164 trunc from 0x99e6b5 to 0x271b0e (0x72cba6 bytes) 000103[0] 1761277289.406994 mapwrite 0x08063b thru 0x090200 (0xfbc6 bytes) 000108[0] 1761277289.445237 mapwrite 0x302134 thru 0x3057a8 (0x3675 bytes) 000110[0] 1761277289.472050 write 0x060556 thru 0x06e612 (0xe0bd bytes) 000111[0] 1761277289.475209 mapread 0x0d1864 thru 0x0d1edb (0x678 bytes) 000114[1] 1761277289.486058 trunc from 0x3057a9 to 0x7c20d2 (0x4bc92a bytes) 000115[0] 1761277289.500037 read 0x3ef30a thru 0x3f6ca1 (0x7998 bytes) 000116[1] 1761277289.532312 trunc from 0x7c20d3 to 0x0778dd (0x74a7f5 bytes) 000118[0] 1761277289.548262 trunc from 0x0778de to 0x2dad1e (0x263441 bytes) 000120[1] 1761277289.565407 trunc from 0x2dad1f to 0x8d45d6 (0x5f98b8 bytes) 000121[1] 1761277289.577483 mapread 0x6bc32e thru 0x6ca5fd (0xe2d0 bytes) 000123[0] 1761277289.590848 read 0x6f378f thru 0x6f9e32 (0x66a4 bytes) 000124[0] 1761277289.599519 trunc from 0x8d45d7 to 0x7a6308 (0x12e2ce bytes) 000125[1] 1761277289.612011 mapread 0x245e94 thru 0x24f04f (0x91bc bytes) 000128[1] 1761277289.626812 mapwrite 0x4bb468 thru 0x4bbc38 (0x7d1 bytes) 000129[0] 1761277289.642041 mapwrite 0x14c09a thru 0x14ea35 (0x299c bytes) 000130[1] 1761277289.668670 trunc from 0x7a6309 to 0x493f1c (0x3123ec bytes) 000132[1] 1761277289.685036 mapwrite 0x61d277 thru 0x6274c1 (0xa24b bytes) 000133[1] 1761277289.707592 mapwrite 0x88d66f thru 0x899dbf (0xc751 bytes) 000135[1] 1761277289.741947 read 0x14c54f thru 0x1546ff (0x81b1 bytes) 000136[0] 1761277289.749030 read 0x2cca94 thru 0x2d99f1 (0xcf5e bytes) 000137[0] 1761277289.757904 trunc from 0x899dc0 to 0x096ad1 (0x8032ee bytes) 000138[1] 1761277289.771129 write 0x4d7092 thru 0x4e5492 (0xe401 bytes) 000139[0] 1761277289.780834 write 0x2b6891 thru 0x2bec2f (0x839f bytes) 000140[0] 1761277289.785926 mapread 0x49f8ef thru 0x4a58e6 (0x5ff8 bytes) 000143[1] 1761277289.798104 mapread 0x3dfe1a thru 0x3ec007 (0xc1ee bytes) 000144[1] 1761277289.824533 read 0x35a11d thru 0x35e0a1 (0x3f85 bytes) 000146[0] 1761277289.831087 trunc from 0x4e5493 to 0x536eb7 (0x51a25 bytes) 000147[1] 1761277289.846496 write 0x6f95a2 thru 0x7035da (0xa039 bytes) 000149[0] 1761277289.855720 read 0x587921 thru 0x58d66b (0x5d4b bytes) 000150[0] 1761277289.865083 read 0x6f6c99 thru 0x6fe008 (0x7370 bytes) 000152[0] 1761277289.870873 mapwrite 0x477895 thru 0x486029 (0xe795 bytes) 000155[1] 1761277289.902758 mapwrite 0x6a3207 thru 0x6ad20d (0xa007 bytes) 000156[0] 1761277289.925326 mapread 0x14498d thru 0x14b8b2 (0x6f26 bytes) 000157[0] 1761277289.935161 read 0x1b64ab thru 0x1c4534 (0xe08a bytes) 000159[1] 1761277289.943508 trunc from 0x7035db to 0x08701c (0x67c5be bytes) 000160[0] 1761277289.951154 read 0x027bfd thru 0x02fb17 (0x7f1b bytes) 000161[1] 1761277289.956753 read 0x04e003 thru 0x04f542 (0x1540 bytes) 000163[0] 1761277289.962408 write 0x00c22c thru 0x00f986 (0x375b bytes) 000166[1] 1761277289.979441 mapread 0x05bac5 thru 0x060835 (0x4d71 bytes) 000167[1] 1761277289.995268 mapwrite 0x067c73 thru 0x071a59 (0x9de7 bytes) 000168[1] 1761277290.027064 mapread 0x0270ba thru 0x02d070 (0x5fb7 bytes) 000171[1] 1761277290.038253 write 0x4e5fdb thru 0x4ec7b3 (0x67d9 bytes) 000172[0] 1761277290.039885 write 0x93d6cd thru 0x948e33 (0xb767 bytes) 000173[1] 1761277290.046377 mapread 0x5c5076 thru 0x5d2917 (0xd8a2 bytes) 000178[1] 1761277290.100860 mapwrite 0x6ddd71 thru 0x6e760b (0x989b bytes) 000179[0] 1761277290.166927 mapwrite 0x427ee2 thru 0x436f1a (0xf039 bytes) 000186[0] 1761277290.208910 write 0x27830e thru 0x283ca7 (0xb99a bytes) 000189[1] 1761277290.224840 read 0x3833b4 thru 0x38ee42 (0xba8f bytes) 000190[0] 1761277290.231826 read 0x3ee2e3 thru 0x3f98b1 (0xb5cf bytes) 000192[0] 1761277290.234107 mapread 0x4e8971 thru 0x4ea9b6 (0x2046 bytes) 000196[0] 1761277290.244999 write 0x29e743 thru 0x2a6283 (0x7b41 bytes) 000198[0] 1761277290.256346 mapwrite 0x82a669 thru 0x82d2ed (0x2c85 bytes) 000199[1] 1761277290.274504 trunc from 0x948e34 to 0x6f927e (0x24fbb5 bytes) 000202[1] 1761277290.292065 trunc from 0x6f927f to 0x496c1e (0x262660 bytes) 000203[0] 1761277290.300546 mapwrite 0x9f18bc thru 0x9fd533 (0xbc78 bytes) 000204[0] 1761277290.330102 trunc from 0x9fd534 to 0x724bbc (0x2d8977 bytes) 000206[1] 1761277290.339071 trunc from 0x724bbd to 0x45bab3 (0x2c9109 bytes) 000207[0] 1761277290.346931 write 0x545e84 thru 0x54cf74 (0x70f1 bytes) 000208[0] 1761277290.353128 trunc from 0x54cf75 to 0x230e22 (0x31c152 bytes) 000209[0] 1761277290.364118 mapread 0x1f2063 thru 0x1fdb82 (0xbb20 bytes) 000210[0] 1761277290.377970 mapwrite 0x1597e2 thru 0x15ab94 (0x13b3 bytes) 000213[0] 1761277290.403755 trunc from 0x230e23 to 0x33d928 (0x10cb06 bytes) 000215[0] 1761277290.414858 mapwrite 0x768719 thru 0x778043 (0xf92b bytes) 000216[1] 1761277290.443652 mapread 0x6026aa thru 0x602b0e (0x465 bytes) 000219[1] 1761277290.454852 read 0x20db1f thru 0x214cdc (0x71be bytes) 000221[1] 1761277290.460940 write 0x0206b8 thru 0x0277ea (0x7133 bytes) 000224[1] 1761277290.469941 read 0x4d3708 thru 0x4df597 (0xbe90 bytes) 000227[1] 1761277290.478267 trunc from 0x778044 to 0x659d20 (0x11e323 bytes) 000230[1] 1761277290.489895 mapread 0x2b6306 thru 0x2b8d94 (0x2a8f bytes) 000232[0] 1761277290.501786 trunc from 0x659d21 to 0x3b5b89 (0x2a4197 bytes) 000233[0] 1761277290.515708 mapread 0x197a18 thru 0x1a76aa (0xfc93 bytes) 000234[0] 1761277290.525471 mapread 0x384e1f thru 0x391278 (0xc45a bytes) 000235[0] 1761277290.535151 mapread 0x2c6d7b thru 0x2c99af (0x2c35 bytes) 000237[0] 1761277290.544999 write 0x662e82 thru 0x672c4e (0xfdcd bytes) 000238[0] 1761277290.551189 trunc from 0x672c4f to 0x5dea60 (0x941ee bytes) truncating to largest ever: 0x9a4ceb 000245[1] 1761277290.564892 trunc from 0x5dea61 to 0x9a4cea (0x3c628a bytes) 000246[1] 1761277290.574426 read 0x44dfad thru 0x452d00 (0x4d54 bytes) 000247[1] 1761277290.581457 write 0x4ac099 thru 0x4ac8fd (0x865 bytes) 000248[1] 1761277290.583079 read 0x58d60d thru 0x58eab4 (0x14a8 bytes) 000249[1] 1761277290.587976 trunc from 0x9a4ceb to 0x5eb56b (0x3b977f bytes) 000253[1] 1761277290.601175 mapwrite 0x351775 thru 0x35a807 (0x9093 bytes) 000254[1] 1761277290.616787 read 0x0a1b92 thru 0x0ad0aa (0xb519 bytes) 000255[1] 1761277290.619757 write 0x2dad5a thru 0x2e4acf (0x9d76 bytes) 000256[0] 1761277290.622440 mapwrite 0x3c69fc thru 0x3ca868 (0x3e6d bytes) 000262[0] 1761277290.667103 mapwrite 0x1ad2ef thru 0x1ba16c (0xce7e bytes) 000263[1] 1761277290.693610 mapwrite 0x84ff6c thru 0x853521 (0x35b6 bytes) 000264[1] 1761277290.721829 write 0x58a7d4 thru 0x592de5 (0x8612 bytes) 000265[1] 1761277290.728040 mapread 0x5f4f26 thru 0x5f81e2 (0x32bd bytes) 000266[0] 1761277290.734537 mapread 0x83a030 thru 0x8451a2 (0xb173 bytes) 000268[0] 1761277290.750421 read 0x0e9686 thru 0x0eb31a (0x1c95 bytes) 000269[1] 1761277290.755351 mapread 0x09cc8f thru 0x0a3fcd (0x733f bytes) 000272[1] 1761277290.766277 mapread 0x5ad2ed thru 0x5b6f8a (0x9c9e bytes) 000273[1] 1761277290.774833 write 0x65c394 thru 0x66b2e9 (0xef56 bytes) 000274[0] 1761277290.778875 read 0x4c1ce0 thru 0x4c3963 (0x1c84 bytes) 000275[0] 1761277290.785085 write 0x22877a thru 0x229ec5 (0x174c bytes) 000278[1] 1761277290.797528 read 0x1eb4c1 thru 0x1f4640 (0x9180 bytes) 000279[1] 1761277290.805033 write 0x5c83c5 thru 0x5d3e9b (0xbad7 bytes) 000280[1] 1761277290.811323 read 0x78d6a2 thru 0x798541 (0xaea0 bytes) 000283[1] 1761277290.818593 trunc from 0x853522 to 0x88dc03 (0x3a6e2 bytes) 000284[0] 1761277290.825748 trunc from 0x88dc04 to 0x07b801 (0x812402 bytes) 000285[1] 1761277290.838591 mapread 0x02e784 thru 0x031b03 (0x3380 bytes) 000288[0] 1761277290.854331 trunc from 0x07b802 to 0x30db0c (0x29230b bytes) 000289[1] 1761277290.865782 mapread 0x0309e2 thru 0x038567 (0x7b86 bytes) 000293[1] 1761277290.882957 write 0x335afd thru 0x33c39d (0x68a1 bytes) 000294[1] 1761277290.889299 write 0x0a875f thru 0x0b27d1 (0xa073 bytes) 000296[1] 1761277290.900787 read 0x327656 thru 0x332ec5 (0xb870 bytes) 000297[0] 1761277290.922569 write 0x1234d3 thru 0x12703d (0x3b6b bytes) 000299[0] 1761277290.935246 read 0x22f88b thru 0x23c59d (0xcd13 bytes) 000301[0] 1761277290.965156 trunc from 0x33c39e to 0x73f24f (0x402eb2 bytes) 000302[0] 1761277290.976181 trunc from 0x73f250 to 0x95e5a9 (0x21f35a bytes) 000306[1] 1761277290.995539 mapread 0x274889 thru 0x27637c (0x1af4 bytes) 000308[1] 1761277291.019714 mapread 0x061fa9 thru 0x0654e8 (0x3540 bytes) 000309[1] 1761277291.027397 mapread 0x60f465 thru 0x617c55 (0x87f1 bytes) 000311[0] 1761277291.065419 trunc from 0x95e5aa to 0x8a62e9 (0xb82c0 bytes) 000312[0] 1761277291.078189 mapread 0x6ab0a3 thru 0x6ae624 (0x3582 bytes) 000314[0] 1761277291.129135 write 0x96cd87 thru 0x96d043 (0x2bd bytes) 000315[0] 1761277291.134954 write 0x7be3ab thru 0x7ccffa (0xec50 bytes) 000317[0] 1761277291.140844 trunc from 0x96d044 to 0x1abb1f (0x7c1524 bytes) 000318[0] 1761277291.156193 trunc from 0x1abb20 to 0x30bfec (0x1604cd bytes) 000319[0] 1761277291.161890 write 0x3aaf96 thru 0x3adc0e (0x2c79 bytes) 000320[1] 1761277291.167716 mapwrite 0x483b01 thru 0x484492 (0x992 bytes) 000321[0] 1761277291.193113 read 0x1da35c thru 0x1e0c9d (0x6942 bytes) 000324[1] 1761277291.237890 mapwrite 0x19bab3 thru 0x1aa00c (0xe55a bytes) 000325[1] 1761277291.282291 read 0x31020c thru 0x311b3d (0x1932 bytes) 000326[1] 1761277291.288402 write 0x84f3da thru 0x85b594 (0xc1bb bytes) 000329[1] 1761277291.298112 read 0x5851f4 thru 0x58d5cd (0x83da bytes) 000330[1] 1761277291.314103 trunc from 0x85b595 to 0x43fa27 (0x41bb6d bytes) 000333[0] 1761277291.331868 read 0x3a134b thru 0x3abcfe (0xa9b4 bytes) 000334[0] 1761277291.338103 trunc from 0x43fa28 to 0x51b2b1 (0xdb88a bytes) 000335[1] 1761277291.348841 read 0x166aaf thru 0x16e44d (0x799f bytes) 000336[0] 1761277291.358723 trunc from 0x51b2b2 to 0x8f08d8 (0x3d5627 bytes) 000338[0] 1761277291.368175 write 0x952d68 thru 0x954f33 (0x21cc bytes) 000339[0] 1761277291.373574 trunc from 0x954f34 to 0x74be74 (0x2090bf bytes) 000341[0] 1761277291.385766 write 0x04ebb6 thru 0x054c22 (0x606d bytes) 000342[1] 1761277291.389346 write 0x99924c thru 0x9a486f (0xb624 bytes) 000344[1] 1761277291.400666 mapread 0x470f30 thru 0x47c3e3 (0xb4b4 bytes) 000345[0] 1761277291.411810 trunc from 0x9a4870 to 0x78c77c (0x2180f3 bytes) 000346[0] 1761277291.426871 mapwrite 0x1fe9d7 thru 0x207b4a (0x9174 bytes) 000347[0] 1761277291.446305 trunc from 0x78c77d to 0x8c6d92 (0x13a616 bytes) 000348[1] 1761277291.455455 read 0x269782 thru 0x269eb1 (0x730 bytes) 000352[0] 1761277291.472476 write 0x43d375 thru 0x43f71b (0x23a7 bytes) 000356[0] 1761277291.480093 write 0x75c5fe thru 0x76b60d (0xf010 bytes) 000358[0] 1761277291.490857 write 0x0ee61e thru 0x0f46a9 (0x608c bytes) 000364[0] 1761277291.514836 read 0x62ca49 thru 0x6328a5 (0x5e5d bytes) 000366[0] 1761277291.517448 mapread 0x27a660 thru 0x280531 (0x5ed2 bytes) 000367[1] 1761277291.522561 trunc from 0x8c6d93 to 0x885ad0 (0x412c2 bytes) 000371[0] 1761277291.549327 trunc from 0x885ad1 to 0x3d2e94 (0x4b2c3c bytes) 000372[1] 1761277291.558229 write 0x7afae1 thru 0x7b2ab3 (0x2fd3 bytes) 000374[1] 1761277291.568015 trunc from 0x7b2ab4 to 0x00589b (0x7ad218 bytes) 000378[1] 1761277291.580688 mapread 0x003110 thru 0x00465b (0x154c bytes) 000379[1] 1761277291.583828 mapwrite 0x84981b thru 0x84ccf4 (0x34da bytes) 000381[1] 1761277291.618158 read 0x84c68d thru 0x84ccf4 (0x668 bytes) 000383[1] 1761277291.624377 mapread 0x37504c thru 0x377d54 (0x2d09 bytes) 000384[1] 1761277291.630446 mapread 0x320eae thru 0x321984 (0xad7 bytes) 000385[1] 1761277291.633650 read 0x54c9ad thru 0x55a312 (0xd966 bytes) 000386[1] 1761277291.636168 trunc from 0x84ccf5 to 0x27decf (0x5cee25 bytes) 000388[0] 1761277291.649902 mapread 0x014aad thru 0x015341 (0x895 bytes) 000389[0] 1761277291.661062 mapread 0x154f5b thru 0x1592c6 (0x436c bytes) 000393[0] 1761277291.673235 mapwrite 0x75dc66 thru 0x768f25 (0xb2c0 bytes) 000396[1] 1761277291.713462 write 0x1f8cba thru 0x1ffc66 (0x6fad bytes) 000397[1] 1761277291.722490 mapread 0x52586f thru 0x5302a8 (0xaa3a bytes) 000398[1] 1761277291.736260 mapwrite 0x07e5ec thru 0x080edd (0x28f2 bytes) 000400[0] 1761277291.760575 write 0x644f7d thru 0x645bfd (0xc81 bytes) 000401[1] 1761277291.770595 mapwrite 0x740d16 thru 0x740f84 (0x26f bytes) 000402[0] 1761277291.788698 read 0x12dfd5 thru 0x135cf5 (0x7d21 bytes) 000403[0] 1761277291.798385 write 0x7deb9e thru 0x7e88ba (0x9d1d bytes) 000404[0] 1761277291.805147 trunc from 0x7e88bb to 0x41e35a (0x3ca560 bytes) 000405[1] 1761277291.817090 trunc from 0x41e35b to 0x2717d6 (0x1acb84 bytes) 000406[0] 1761277291.829167 mapread 0x000724 thru 0x00e9c6 (0xe2a3 bytes) 000407[0] 1761277291.841765 mapwrite 0x533e40 thru 0x5397c9 (0x598a bytes) 000408[0] 1761277291.868736 mapwrite 0x94cdcd thru 0x954bb7 (0x7deb bytes) 000409[0] 1761277291.896572 trunc from 0x954bb8 to 0x037b5c (0x91d05b bytes) 000410[0] 1761277291.906164 read 0x004510 thru 0x0124a3 (0xdf94 bytes) 000411[1] 1761277291.914788 write 0x751db9 thru 0x75fe0f (0xe057 bytes) 000412[1] 1761277291.925818 mapwrite 0x45d744 thru 0x465f83 (0x8840 bytes) 000413[1] 1761277291.952419 mapread 0x1538b2 thru 0x162c0c (0xf35b bytes) 000416[1] 1761277291.974878 mapwrite 0x8ee7ba thru 0x8f405b (0x58a2 bytes) 000419[0] 1761277292.002534 trunc from 0x8f405c to 0x828a04 (0xcb657 bytes) 000421[0] 1761277292.016055 mapread 0x227982 thru 0x22efed (0x766c bytes) 000423[0] 1761277292.054867 read 0x125dde thru 0x129eba (0x40dd bytes) 000424[0] 1761277292.055251 write 0x37cffa thru 0x3834d8 (0x64df bytes) 000429[1] 1761277292.070268 trunc from 0x828a05 to 0x198365 (0x69069f bytes) 000431[1] 1761277292.097561 write 0x5a76bc thru 0x5af19a (0x7adf bytes) 000434[0] 1761277292.102559 mapread 0x1f356a thru 0x1f9f5f (0x69f6 bytes) 000437[0] 1761277292.141275 write 0x9712d1 thru 0x9732e3 (0x2013 bytes) 000439[0] 1761277292.153124 mapread 0x1336ad thru 0x134f00 (0x1854 bytes) 000442[0] 1761277292.169195 read 0x7be08d thru 0x7ca9da (0xc94e bytes) 000447[1] 1761277292.192257 mapread 0x8b21dd thru 0x8b318c (0xfb0 bytes) 000448[1] 1761277292.200916 read 0x1e33ea thru 0x1f0b16 (0xd72d bytes) 000451[0] 1761277292.204095 read 0x17cedf thru 0x18a3ee (0xd510 bytes) 000452[1] 1761277292.207672 mapread 0x6e7df0 thru 0x6e7e38 (0x49 bytes) 000453[1] 1761277292.209272 read 0x72f9da thru 0x735942 (0x5f69 bytes) 000455[1] 1761277292.218043 read 0x808467 thru 0x817795 (0xf32f bytes) 000456[1] 1761277292.221179 mapwrite 0x5a736e thru 0x5a804f (0xce2 bytes) 000460[1] 1761277292.247887 trunc from 0x9732e4 to 0x91f6ec (0x53bf7 bytes) 000464[0] 1761277292.264847 trunc from 0x91f6ed to 0x157005 (0x7c86e7 bytes) 000466[1] 1761277292.280500 read 0x0a332d thru 0x0aaa02 (0x76d6 bytes) 000468[0] 1761277292.297391 write 0x43df8c thru 0x43e613 (0x688 bytes) 000469[1] 1761277292.298794 read 0x2de843 thru 0x2e2705 (0x3ec3 bytes) 000471[1] 1761277292.308365 write 0x1cbe99 thru 0x1d09a6 (0x4b0e bytes) 000472[0] 1761277292.316604 write 0x464958 thru 0x4662a7 (0x1950 bytes) 000477[0] 1761277292.335520 mapwrite 0x4e2438 thru 0x4e65bc (0x4185 bytes) 000479[1] 1761277292.368332 mapwrite 0x4d56dd thru 0x4d9fc2 (0x48e6 bytes) 000480[1] 1761277292.392947 mapwrite 0x0c2dd3 thru 0x0c8f8f (0x61bd bytes) 000487[1] 1761277292.430539 mapwrite 0x31f20d thru 0x32691f (0x7713 bytes) truncating to largest ever: 0x9bf488 000488[1] 1761277292.450447 trunc from 0x4e65bd to 0x9bf487 (0x4d8ecb bytes) 000491[1] 1761277292.459022 trunc from 0x9bf488 to 0x284cb2 (0x73a7d5 bytes) 000492[0] 1761277292.467883 write 0x5f0e0c thru 0x5ffd7f (0xef74 bytes) 000493[1] 1761277292.476807 read 0x4f6f1f thru 0x506c0a (0xfcec bytes) 000494[1] 1761277292.486059 read 0x21dab1 thru 0x21e29e (0x7ee bytes) 000497[0] 1761277292.495695 read 0x115781 thru 0x11ca5b (0x72db bytes) 000499[1] 1761277292.502111 mapwrite 0x903396 thru 0x90e2ab (0xaf16 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7292 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004[1] 1761277292.799018 mapwrite 0x561000 thru 0x56ed12 (0xdd13 bytes) 000006[0] 1761277292.834057 mapread 0x14f000 thru 0x150bae (0x1baf bytes) 000008[0] 1761277292.843186 mapwrite 0x0d1000 thru 0x0dfef4 (0xeef5 bytes) 000009[1] 1761277292.867364 mapwrite 0x84a000 thru 0x84a936 (0x937 bytes) 000011[1] 1761277292.888874 read 0x2c8000 thru 0x2d0c1c (0x8c1d bytes) 000012[1] 1761277292.893926 write 0x329000 thru 0x3297de (0x7df bytes) 000015[1] 1761277292.901418 mapwrite 0x239000 thru 0x246cd7 (0xdcd8 bytes) 000016[1] 1761277292.927844 read 0x4f3000 thru 0x500dcc (0xddcd bytes) 000021[1] 1761277292.939078 mapwrite 0x967000 thru 0x9691f5 (0x21f6 bytes) 000022[1] 1761277292.966722 mapwrite 0x7cc000 thru 0x7da8dd (0xe8de bytes) 000023[0] 1761277292.988975 mapread 0x6af000 thru 0x6b1bbd (0x2bbe bytes) 000026[0] 1761277293.003310 write 0x621000 thru 0x62a1e4 (0x91e5 bytes) 000030[1] 1761277293.015011 mapread 0x1c6000 thru 0x1cf99c (0x999d bytes) truncating to largest ever: 0x55d396 000032[1] 1761277293.030314 trunc from 0x9691f6 to 0x55d395 (0x40be60 bytes) 000033[1] 1761277293.039078 trunc from 0x55d396 to 0x26f432 (0x2edf63 bytes) 000034[1] 1761277293.042812 trunc from 0x26f433 to 0x104ade (0x16a954 bytes) 000036[1] 1761277293.053462 read 0x055000 thru 0x0566b8 (0x16b9 bytes) 000039[1] 1761277293.073874 mapwrite 0x9b9000 thru 0x9c1d15 (0x8d16 bytes) 000040[0] 1761277293.190796 mapread 0x867000 thru 0x875bc9 (0xebca bytes) 000044[1] 1761277293.220032 mapread 0x039000 thru 0x03a1f1 (0x11f2 bytes) 000045[0] 1761277293.223410 mapwrite 0x3ba000 thru 0x3c70bf (0xd0c0 bytes) 000047[0] 1761277293.250611 write 0x606000 thru 0x60b8e8 (0x58e9 bytes) 000049[0] 1761277293.256566 mapread 0x73b000 thru 0x74a3a7 (0xf3a8 bytes) 000051[1] 1761277293.270476 read 0x9bb000 thru 0x9c17a2 (0x67a3 bytes) 000052[0] 1761277293.276823 write 0x712000 thru 0x7171e9 (0x51ea bytes) truncating to largest ever: 0x90ae4a 000053[1] 1761277293.284057 trunc from 0x9c1d16 to 0x90ae49 (0xb6ecc bytes) 000055[1] 1761277293.299597 write 0x768000 thru 0x76ad16 (0x2d17 bytes) 000056[1] 1761277293.304654 read 0x05d000 thru 0x068c24 (0xbc25 bytes) 000057[1] 1761277293.310644 write 0x0f7000 thru 0x104a50 (0xda51 bytes) 000063[0] 1761277293.320660 trunc from 0x90ae4a to 0x3cd4d6 (0x53d973 bytes) 000065[1] 1761277293.330223 mapread 0x061000 thru 0x067f6d (0x6f6e bytes) 000066[1] 1761277293.338655 trunc from 0x3cd4d7 to 0x8a44f7 (0x4d7021 bytes) 000072[1] 1761277293.356120 read 0x48b000 thru 0x494c41 (0x9c42 bytes) 000073[0] 1761277293.358745 mapread 0x4e6000 thru 0x4e8bde (0x2bdf bytes) 000074[0] 1761277293.367532 trunc from 0x8a44f8 to 0x7c86c5 (0xdbe32 bytes) 000080[1] 1761277293.382157 read 0x0a9000 thru 0x0acdb3 (0x3db4 bytes) 000081[1] 1761277293.385986 write 0x8ab000 thru 0x8ae9a0 (0x39a1 bytes) 000082[0] 1761277293.388699 mapwrite 0x8f9000 thru 0x90067a (0x767b bytes) 000084[0] 1761277293.417128 write 0x521000 thru 0x530165 (0xf166 bytes) 000086[0] 1761277293.425862 mapread 0x247000 thru 0x24b25d (0x425e bytes) 000089[1] 1761277293.434521 mapread 0x53e000 thru 0x54c5ec (0xe5ed bytes) 000090[1] 1761277293.451282 trunc from 0x90067b to 0x35ab50 (0x5a5b2a bytes) 000092[1] 1761277293.466354 read 0x04a000 thru 0x052102 (0x8103 bytes) 000093[1] 1761277293.489104 read 0x189000 thru 0x18ceb4 (0x3eb5 bytes) 000094[0] 1761277293.492701 write 0x672000 thru 0x67b98a (0x998b bytes) 000095[0] 1761277293.501555 trunc from 0x67b98b to 0x564469 (0x117521 bytes) 000096[0] 1761277293.506203 mapwrite 0x6f7000 thru 0x705d49 (0xed4a bytes) 000100[1] 1761277293.530029 trunc from 0x705d4a to 0x778b37 (0x72dee bytes) 000101[1] 1761277293.538965 write 0x3c7000 thru 0x3d1711 (0xa712 bytes) 000103[1] 1761277293.568265 mapread 0x6ff000 thru 0x706202 (0x7203 bytes) 000105[0] 1761277293.601199 write 0x480000 thru 0x48a3d0 (0xa3d1 bytes) 000109[0] 1761277293.625385 read 0x02a000 thru 0x037be0 (0xdbe1 bytes) 000111[1] 1761277293.631957 read 0x3ac000 thru 0x3b3d11 (0x7d12 bytes) 000114[0] 1761277293.638904 trunc from 0x778b38 to 0x4dc632 (0x29c505 bytes) 000118[0] 1761277293.652345 read 0x3c5000 thru 0x3c7b7a (0x2b7b bytes) 000122[0] 1761277293.662861 write 0x063000 thru 0x06593b (0x293c bytes) 000123[0] 1761277293.667590 trunc from 0x4dc633 to 0x8fc941 (0x42030f bytes) 000130[1] 1761277293.687854 read 0x51a000 thru 0x51c115 (0x2116 bytes) 000131[1] 1761277293.693350 mapwrite 0x4a5000 thru 0x4b2020 (0xd021 bytes) 000132[1] 1761277293.718486 mapread 0x2fd000 thru 0x301be2 (0x4be3 bytes) 000133[0] 1761277293.728477 mapread 0x70c000 thru 0x70ee50 (0x2e51 bytes) 000135[1] 1761277293.740787 write 0x473000 thru 0x4754df (0x24e0 bytes) 000136[0] 1761277293.748777 mapread 0x462000 thru 0x467e55 (0x5e56 bytes) 000139[1] 1761277293.760582 write 0x4e0000 thru 0x4ee44d (0xe44e bytes) 000146[0] 1761277293.780798 read 0x787000 thru 0x78a17c (0x317d bytes) 000147[0] 1761277293.787394 mapread 0x378000 thru 0x37ce0c (0x4e0d bytes) 000149[1] 1761277293.792645 read 0x713000 thru 0x722165 (0xf166 bytes) 000150[1] 1761277293.795222 trunc from 0x8fc942 to 0x1c0d84 (0x73bbbd bytes) 000151[1] 1761277293.802428 read 0x195000 thru 0x1a14e9 (0xc4ea bytes) 000152[0] 1761277293.818422 write 0x438000 thru 0x43cbdb (0x4bdc bytes) 000153[1] 1761277293.823357 trunc from 0x43cbdc to 0x0d64a5 (0x366736 bytes) 000158[0] 1761277293.839795 read 0x056000 thru 0x05cb3a (0x6b3b bytes) 000159[1] 1761277293.848756 write 0x626000 thru 0x62c84c (0x684d bytes) 000163[1] 1761277293.863037 mapwrite 0x4be000 thru 0x4c7e97 (0x9e98 bytes) 000164[0] 1761277293.898132 read 0x1e9000 thru 0x1f4e57 (0xbe58 bytes) 000169[0] 1761277293.908197 write 0x20a000 thru 0x210103 (0x6104 bytes) 000170[1] 1761277293.915042 read 0x4b5000 thru 0x4b76a6 (0x26a7 bytes) 000173[0] 1761277293.921537 read 0x476000 thru 0x480684 (0xa685 bytes) 000174[0] 1761277293.923695 write 0x098000 thru 0x0a4835 (0xc836 bytes) 000175[0] 1761277293.927848 mapread 0x53a000 thru 0x541610 (0x7611 bytes) 000177[0] 1761277293.932504 write 0x042000 thru 0x04d366 (0xb367 bytes) 000184[0] 1761277293.953379 mapread 0x2dc000 thru 0x2df23e (0x323f bytes) 000186[0] 1761277293.963669 mapread 0x074000 thru 0x074650 (0x651 bytes) 000190[1] 1761277293.972207 mapwrite 0x865000 thru 0x86871c (0x371d bytes) 000191[0] 1761277294.002478 mapread 0x356000 thru 0x35957c (0x357d bytes) truncating to largest ever: 0x9ceb12 000193[1] 1761277294.013573 trunc from 0x86871d to 0x9ceb11 (0x1663f5 bytes) 000194[1] 1761277294.023546 read 0x661000 thru 0x66aa02 (0x9a03 bytes) 000195[1] 1761277294.029200 mapwrite 0x222000 thru 0x229343 (0x7344 bytes) 000197[0] 1761277294.050913 trunc from 0x9ceb12 to 0x2e841d (0x6e66f4 bytes) 000201[1] 1761277294.065335 read 0x2bc000 thru 0x2c8f41 (0xcf42 bytes) 000203[0] 1761277294.087469 read 0x0c1000 thru 0x0cb5f5 (0xa5f6 bytes) 000205[0] 1761277294.104986 mapread 0x2be000 thru 0x2c1c08 (0x3c09 bytes) 000206[0] 1761277294.108038 write 0x7d9000 thru 0x7dfc84 (0x6c85 bytes) 000208[1] 1761277294.125994 read 0x714000 thru 0x720d73 (0xcd74 bytes) 000210[0] 1761277294.132356 mapread 0x781000 thru 0x7874b3 (0x64b4 bytes) 000213[0] 1761277294.166649 trunc from 0x7dfc85 to 0x4ae514 (0x331770 bytes) 000214[0] 1761277294.177544 write 0x2a1000 thru 0x2a48ba (0x38bb bytes) 000216[1] 1761277294.186020 mapwrite 0x79b000 thru 0x79f8d9 (0x48da bytes) 000217[0] 1761277294.211686 mapread 0x0cf000 thru 0x0d3bda (0x4bdb bytes) 000218[1] 1761277294.221003 read 0x5f3000 thru 0x6001d8 (0xd1d9 bytes) 000219[1] 1761277294.225342 mapwrite 0x5dc000 thru 0x5e24d4 (0x64d5 bytes) 000221[1] 1761277294.246721 trunc from 0x79f8da to 0x6c1ce1 (0xddbf8 bytes) 000223[1] 1761277294.255055 read 0x238000 thru 0x239061 (0x1062 bytes) 000228[0] 1761277294.265643 write 0x498000 thru 0x49a527 (0x2528 bytes) 000231[1] 1761277294.278384 write 0x65b000 thru 0x66885a (0xd85b bytes) 000232[0] 1761277294.283493 mapread 0x5bf000 thru 0x5ce742 (0xf743 bytes) 000233[1] 1761277294.308054 mapwrite 0x9af000 thru 0x9b1453 (0x2454 bytes) 000234[1] 1761277294.332058 write 0x79e000 thru 0x7a6891 (0x8892 bytes) 000235[1] 1761277294.334159 read 0x5a4000 thru 0x5ad090 (0x9091 bytes) 000237[1] 1761277294.344123 read 0x562000 thru 0x5638ee (0x18ef bytes) truncating to largest ever: 0x9e2492 000238[1] 1761277294.349736 trunc from 0x9b1454 to 0x9e2491 (0x3103e bytes) 000243[0] 1761277294.362786 read 0x81f000 thru 0x82199e (0x299f bytes) 000244[0] 1761277294.369184 write 0x164000 thru 0x165a31 (0x1a32 bytes) 000245[0] 1761277294.376027 mapwrite 0x535000 thru 0x53d81b (0x881c bytes) 000246[0] 1761277294.398359 mapwrite 0x42e000 thru 0x43c035 (0xe036 bytes) 000248[0] 1761277294.425151 write 0x124000 thru 0x132412 (0xe413 bytes) 000249[0] 1761277294.427132 trunc from 0x9e2492 to 0x294e5e (0x74d633 bytes) 000250[1] 1761277294.432176 mapread 0x19d000 thru 0x1a42e5 (0x72e6 bytes) 000257[0] 1761277294.447185 mapread 0x251000 thru 0x257e8c (0x6e8d bytes) 000258[0] 1761277294.452673 write 0x2d0000 thru 0x2df4c6 (0xf4c7 bytes) 000259[1] 1761277294.459492 write 0x1c2000 thru 0x1c9463 (0x7464 bytes) 000261[1] 1761277294.471527 read 0x1a0000 thru 0x1ac807 (0xc808 bytes) 000262[1] 1761277294.496638 mapread 0x108000 thru 0x1142d1 (0xc2d2 bytes) 000263[1] 1761277294.500281 mapread 0x2bd000 thru 0x2c13f8 (0x43f9 bytes) 000266[0] 1761277294.504078 mapread 0x1f5000 thru 0x20165b (0xc65c bytes) 000269[0] 1761277294.516954 trunc from 0x2df4c7 to 0x19039d (0x14f129 bytes) 000270[1] 1761277294.526826 mapread 0x0dc000 thru 0x0e7914 (0xb915 bytes) 000275[1] 1761277294.546027 trunc from 0x19039e to 0x0e7e5e (0xa853f bytes) 000276[1] 1761277294.555974 mapread 0x0da000 thru 0x0ddd17 (0x3d18 bytes) 000278[0] 1761277294.565885 mapread 0x029000 thru 0x037a0d (0xea0e bytes) 000283[1] 1761277294.578436 mapwrite 0x4bf000 thru 0x4cbec1 (0xcec2 bytes) 000284[0] 1761277294.617281 mapread 0x3b1000 thru 0x3bf415 (0xe416 bytes) 000285[0] 1761277294.630504 mapread 0x2d5000 thru 0x2e4bc0 (0xfbc1 bytes) 000286[0] 1761277294.634542 trunc from 0x4cbec2 to 0x41f364 (0xacb5d bytes) 000287[0] 1761277294.641039 write 0x653000 thru 0x656005 (0x3006 bytes) 000289[1] 1761277294.648890 write 0x695000 thru 0x696902 (0x1903 bytes) 000290[1] 1761277294.655683 mapwrite 0x6ec000 thru 0x6ef0e0 (0x30e1 bytes) 000291[1] 1761277294.692514 trunc from 0x6ef0e1 to 0x568e24 (0x1862bc bytes) 000292[1] 1761277294.701610 write 0x9cd000 thru 0x9d7140 (0xa141 bytes) 000293[1] 1761277294.705684 write 0x14f000 thru 0x14f45c (0x45d bytes) 000296[0] 1761277294.714004 mapwrite 0x08b000 thru 0x091ec3 (0x6ec4 bytes) 000297[1] 1761277294.747416 write 0x4a3000 thru 0x4b2a78 (0xfa79 bytes) 000300[0] 1761277294.759305 read 0x20c000 thru 0x21b018 (0xf019 bytes) 000302[0] 1761277294.767429 mapwrite 0x86c000 thru 0x87985b (0xd85c bytes) 000303[0] 1761277294.796428 mapread 0x3b6000 thru 0x3be338 (0x8339 bytes) 000304[0] 1761277294.800976 mapwrite 0x7cf000 thru 0x7cf6f9 (0x6fa bytes) 000305[1] 1761277294.812878 read 0x3df000 thru 0x3e806f (0x9070 bytes) 000306[0] 1761277294.818540 mapwrite 0x282000 thru 0x28b7a0 (0x97a1 bytes) 000308[1] 1761277294.839822 trunc from 0x9d7141 to 0x5fba85 (0x3db6bb bytes) 000309[0] 1761277294.847689 read 0x116000 thru 0x1239f9 (0xd9fa bytes) 000310[1] 1761277294.853574 write 0x242000 thru 0x24489d (0x289e bytes) 000314[0] 1761277294.861342 read 0x4a7000 thru 0x4aa305 (0x3306 bytes) 000315[1] 1761277294.866067 read 0x3d7000 thru 0x3db259 (0x425a bytes) 000317[1] 1761277294.869396 write 0x3da000 thru 0x3e70d3 (0xd0d4 bytes) 000318[1] 1761277294.874324 write 0x386000 thru 0x391c54 (0xbc55 bytes) 000319[1] 1761277294.878862 write 0x130000 thru 0x131380 (0x1381 bytes) 000323[1] 1761277294.886696 mapwrite 0x48b000 thru 0x48f179 (0x417a bytes) 000324[0] 1761277294.906433 write 0x645000 thru 0x64945f (0x4460 bytes) 000325[0] 1761277294.911957 mapwrite 0x5d5000 thru 0x5e12a7 (0xc2a8 bytes) 000327[0] 1761277294.934242 mapwrite 0x4c6000 thru 0x4c87fb (0x27fc bytes) 000331[1] 1761277294.954354 mapread 0x147000 thru 0x14f7d9 (0x87da bytes) 000335[1] 1761277294.964133 mapwrite 0x198000 thru 0x1a7fc9 (0xffca bytes) 000336[0] 1761277294.983097 mapread 0x548000 thru 0x5539fc (0xb9fd bytes) 000337[0] 1761277294.992734 mapread 0x32b000 thru 0x32d51a (0x251b bytes) 000338[0] 1761277295.001677 mapwrite 0x467000 thru 0x474e45 (0xde46 bytes) 000339[0] 1761277295.027702 mapwrite 0x0ca000 thru 0x0d4c6f (0xac70 bytes) 000341[1] 1761277295.046929 trunc from 0x649460 to 0x65de83 (0x14a24 bytes) 000344[0] 1761277295.057419 mapread 0x336000 thru 0x3368e1 (0x8e2 bytes) 000346[0] 1761277295.063138 read 0x174000 thru 0x17cffe (0x8fff bytes) 000347[0] 1761277295.065091 trunc from 0x65de84 to 0x8ccc75 (0x26edf2 bytes) 000350[0] 1761277295.075939 trunc from 0x8ccc76 to 0x744efd (0x187d78 bytes) 000351[1] 1761277295.079961 trunc from 0x744efe to 0x4708bb (0x2d4642 bytes) 000352[1] 1761277295.088654 write 0x728000 thru 0x72d836 (0x5837 bytes) 000355[0] 1761277295.094202 mapread 0x0c9000 thru 0x0cef2c (0x5f2d bytes) 000356[0] 1761277295.104866 mapread 0x0af000 thru 0x0b0ab0 (0x1ab1 bytes) 000358[0] 1761277295.107662 mapread 0x190000 thru 0x192791 (0x2792 bytes) 000361[0] 1761277295.110510 mapwrite 0x48e000 thru 0x49de2b (0xfe2c bytes) 000364[1] 1761277295.145692 mapread 0x6c3000 thru 0x6c7794 (0x4795 bytes) 000370[1] 1761277295.158946 trunc from 0x72d837 to 0x09a363 (0x6934d3 bytes) 000372[0] 1761277295.167043 mapread 0x058000 thru 0x0642fa (0xc2fb bytes) 000378[0] 1761277295.180129 mapwrite 0x49b000 thru 0x4a254f (0x7550 bytes) 000381[1] 1761277295.210241 trunc from 0x4a2550 to 0x162427 (0x340128 bytes) 000382[1] 1761277295.216665 mapread 0x0b7000 thru 0x0b7e01 (0xe02 bytes) 000383[1] 1761277295.222384 write 0x8bd000 thru 0x8bda74 (0xa75 bytes) 000389[0] 1761277295.231659 trunc from 0x8bda75 to 0x20e801 (0x6af273 bytes) 000394[0] 1761277295.249725 trunc from 0x20e802 to 0x79d328 (0x58eb27 bytes) 000397[0] 1761277295.262264 write 0x6ed000 thru 0x6efddb (0x2ddc bytes) 000402[0] 1761277295.274202 mapwrite 0x203000 thru 0x211300 (0xe301 bytes) 000403[0] 1761277295.294035 mapread 0x40e000 thru 0x4106c7 (0x26c8 bytes) 000405[0] 1761277295.299507 write 0x602000 thru 0x60bd7d (0x9d7e bytes) 000406[1] 1761277295.304959 write 0x7df000 thru 0x7ea8d4 (0xb8d5 bytes) 000408[0] 1761277295.314586 trunc from 0x7ea8d5 to 0x1b6bfb (0x633cd9 bytes) 000412[1] 1761277295.330219 mapread 0x08d000 thru 0x096d74 (0x9d75 bytes) 000414[1] 1761277295.336956 mapread 0x099000 thru 0x09ac9d (0x1c9e bytes) 000416[1] 1761277295.339013 write 0x2c1000 thru 0x2d035b (0xf35c bytes) 000417[1] 1761277295.344468 write 0x232000 thru 0x2350b6 (0x30b7 bytes) 000418[0] 1761277295.348800 trunc from 0x2d035c to 0x8c5b65 (0x5f580a bytes) 000419[0] 1761277295.357823 read 0x6d2000 thru 0x6e06bb (0xe6bc bytes) 000420[1] 1761277295.363783 trunc from 0x8c5b66 to 0x79bbb2 (0x129fb3 bytes) 000421[0] 1761277295.369389 trunc from 0x79bbb3 to 0x5ac731 (0x1ef481 bytes) 000423[1] 1761277295.378758 trunc from 0x5ac732 to 0x16263f (0x44a0f2 bytes) 000424[0] 1761277295.384803 write 0x294000 thru 0x2a05c2 (0xc5c3 bytes) 000425[0] 1761277295.390562 write 0x455000 thru 0x4623d8 (0xd3d9 bytes) 000427[0] 1761277295.396319 mapread 0x3c9000 thru 0x3cc679 (0x367a bytes) 000431[1] 1761277295.407532 read 0x24e000 thru 0x2501fd (0x21fe bytes) 000432[1] 1761277295.414564 trunc from 0x4623d9 to 0x1ffb06 (0x2628d2 bytes) 000434[0] 1761277295.425210 read 0x031000 thru 0x03c0ea (0xb0eb bytes) 000436[0] 1761277295.444522 read 0x0e4000 thru 0x0f13a4 (0xd3a5 bytes) 000438[1] 1761277295.446040 trunc from 0x1ffb07 to 0x420a61 (0x220f5b bytes) 000441[1] 1761277295.461090 read 0x292000 thru 0x295bd6 (0x3bd7 bytes) 000442[0] 1761277295.463141 read 0x1ba000 thru 0x1bf218 (0x5219 bytes) 000444[1] 1761277295.467985 mapread 0x1f4000 thru 0x1f7925 (0x3926 bytes) 000447[0] 1761277295.472468 mapread 0x39c000 thru 0x3a586c (0x986d bytes) 000448[1] 1761277295.476844 trunc from 0x420a62 to 0x894bf5 (0x474194 bytes) 000449[0] 1761277295.488126 read 0x053000 thru 0x05f492 (0xc493 bytes) 000452[1] 1761277295.494694 mapwrite 0x4a6000 thru 0x4b220f (0xc210 bytes) 000453[0] 1761277295.514143 mapwrite 0x829000 thru 0x83733e (0xe33f bytes) 000454[1] 1761277295.540670 mapwrite 0x652000 thru 0x654c9f (0x2ca0 bytes) 000463[0] 1761277295.573448 read 0x4b9000 thru 0x4bba68 (0x2a69 bytes) 000464[0] 1761277295.577387 trunc from 0x894bf6 to 0x9b6ec0 (0x1222cb bytes) 000466[1] 1761277295.587796 read 0x64d000 thru 0x652654 (0x5655 bytes) 000467[1] 1761277295.592794 mapwrite 0x7cf000 thru 0x7d0347 (0x1348 bytes) 000478[0] 1761277295.624968 mapread 0x96c000 thru 0x97b7c6 (0xf7c7 bytes) 000480[0] 1761277295.635923 trunc from 0x9b6ec1 to 0x484dc3 (0x5320fd bytes) 000482[1] 1761277295.648352 trunc from 0x484dc4 to 0x8f61f9 (0x471436 bytes) 000484[0] 1761277295.654503 read 0x156000 thru 0x159c18 (0x3c19 bytes) 000487[0] 1761277295.660156 trunc from 0x8f61fa to 0x429205 (0x4ccff4 bytes) 000490[0] 1761277295.673875 mapwrite 0x470000 thru 0x473c6e (0x3c6f bytes) 000492[0] 1761277295.694122 read 0x059000 thru 0x061684 (0x8685 bytes) 000493[0] 1761277295.696891 mapwrite 0x30d000 thru 0x30e75f (0x1760 bytes) 000494[1] 1761277295.708467 read 0x409000 thru 0x40bf9b (0x2f9c bytes) 000496[1] 1761277295.713530 mapread 0x38b000 thru 0x393340 (0x8341 bytes) 000498[1] 1761277295.719088 read 0x389000 thru 0x39073b (0x773c bytes) 000499[0] 1761277295.719351 mapwrite 0x89f000 thru 0x8a9fb9 (0xafba bytes) 000500[1] 1761277295.751606 mapwrite 0x1dd000 thru 0x1e8e4f (0xbe50 bytes) All operations completed A-OK! PASS 16a (10s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 23:41:37 (1761277297) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0842017 s, 99.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0825432 s, 102 MB/s Chance of close/open is 1 in 50 Seed set to 7298 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read 000009[0] 1761277298.259427 write 0x34306c thru 0x34350d (0x4a2 bytes) 000010[1] 1761277298.267003 write 0x0b6af1 thru 0x0b701c (0x52c bytes) truncating to largest ever: 0x488e72 000012[1] 1761277298.277562 trunc from 0x34350e to 0x488e71 (0x145964 bytes) 000013[0] 1761277298.284415 write 0x114f4d thru 0x1155e8 (0x69c bytes) 000016[1] 1761277298.303811 mapread 0x085920 thru 0x085cbc (0x39d bytes) 000019[0] 1761277298.319302 trunc from 0x488e72 to 0x40c252 (0x7cc1f bytes) 000020[1] 1761277298.332192 write 0x568ffa thru 0x56a0ec (0x10f3 bytes) 000021[1] 1761277298.337751 trunc from 0x56a0ed to 0x07c3ff (0x4edced bytes) 000024[0] 1761277298.355596 trunc from 0x07c400 to 0x129a0d (0xad60e bytes) 000025[0] 1761277298.368187 mapwrite 0x35c099 thru 0x35e06e (0x1fd6 bytes) 000027[1] 1761277298.399373 write 0x561d8c thru 0x56344d (0x16c2 bytes) 000028[0] 1761277298.406006 write 0x206c5d thru 0x207f08 (0x12ac bytes) 000030[0] 1761277298.414074 mapwrite 0x420ebb thru 0x42229b (0x13e1 bytes) 000032[1] 1761277298.436687 write 0x24b742 thru 0x24d0e0 (0x199f bytes) 000033[1] 1761277298.445216 write 0x86fe2f thru 0x86fe6d (0x3f bytes) 000035[1] 1761277298.452482 read 0x5c22da thru 0x5c2729 (0x450 bytes) 000036[0] 1761277298.456632 mapwrite 0x74d99d thru 0x74dce4 (0x348 bytes) 000037[1] 1761277298.478357 write 0x51c543 thru 0x51c7dc (0x29a bytes) 000040[0] 1761277298.482630 read 0x050308 thru 0x050bf7 (0x8f0 bytes) 000041[0] 1761277298.488888 write 0x6116d3 thru 0x612785 (0x10b3 bytes) 000042[1] 1761277298.494605 read 0x3f0053 thru 0x3f0ae7 (0xa95 bytes) truncating to largest ever: 0x4fd74a 000043[1] 1761277298.499884 trunc from 0x86fe6e to 0x4fd749 (0x372724 bytes) 000046[1] 1761277298.510406 mapread 0x0a19b6 thru 0x0a2b87 (0x11d2 bytes) 000047[1] 1761277298.517821 mapwrite 0x175e19 thru 0x17719b (0x1383 bytes) 000048[0] 1761277298.534301 mapread 0x03b7e0 thru 0x03b9c3 (0x1e4 bytes) 000050[1] 1761277298.543718 mapread 0x2b0d0d thru 0x2b2322 (0x1616 bytes) 000052[0] 1761277298.557479 write 0x99c710 thru 0x99d4b1 (0xda2 bytes) 000053[1] 1761277298.565333 write 0x9df2d2 thru 0x9e024c (0xf7b bytes) 000054[0] 1761277298.573544 write 0x71a5e0 thru 0x71a8fe (0x31f bytes) 000056[1] 1761277298.582779 mapwrite 0x41d58c thru 0x41e0ce (0xb43 bytes) truncating to largest ever: 0x7b7921 000058[0] 1761277298.610669 trunc from 0x9e024d to 0x7b7920 (0x22892c bytes) 000059[1] 1761277298.624408 mapread 0x6ad00f thru 0x6ad9c9 (0x9bb bytes) 000061[0] 1761277298.633751 write 0x2c52c3 thru 0x2c5e26 (0xb64 bytes) 000063[1] 1761277298.639554 write 0x6b661c thru 0x6b7a79 (0x145e bytes) 000065[1] 1761277298.654456 write 0x26d23e thru 0x26e623 (0x13e6 bytes) 000067[0] 1761277298.664927 mapread 0x673780 thru 0x673b78 (0x3f9 bytes) 000068[1] 1761277298.673316 mapread 0x271df7 thru 0x272269 (0x473 bytes) 000069[1] 1761277298.679196 write 0x42cc3b thru 0x42dea6 (0x126c bytes) 000073[1] 1761277298.692818 mapwrite 0x5f541b thru 0x5f7302 (0x1ee8 bytes) 000074[1] 1761277298.708749 read 0x69b5d3 thru 0x69d0e0 (0x1b0e bytes) 000075[0] 1761277298.710944 mapread 0x018909 thru 0x019ea3 (0x159b bytes) 000076[1] 1761277298.719162 trunc from 0x7b7921 to 0x70123a (0xb66e6 bytes) 000077[1] 1761277298.729558 mapwrite 0x627c73 thru 0x62955a (0x18e8 bytes) 000079[0] 1761277298.746461 read 0x4d18d8 thru 0x4d321e (0x1947 bytes) 000080[1] 1761277298.752814 read 0x12619a thru 0x1271a4 (0x100b bytes) 000082[0] 1761277298.759817 mapwrite 0x73346e thru 0x734a90 (0x1623 bytes) 000083[1] 1761277298.779800 mapwrite 0x9d4291 thru 0x9d5a30 (0x17a0 bytes) 000085[0] 1761277298.808680 trunc from 0x9d5a31 to 0x476a00 (0x55f030 bytes) 000086[0] 1761277298.819529 mapwrite 0x2f0ffd thru 0x2f24c0 (0x14c4 bytes) 000088[1] 1761277298.834854 write 0x7565c1 thru 0x756c7f (0x6bf bytes) 000089[1] 1761277298.840420 mapwrite 0x00869a thru 0x008f2b (0x892 bytes) 000090[1] 1761277298.853116 mapwrite 0x64f058 thru 0x64f6f6 (0x69f bytes) 000091[0] 1761277298.863886 mapwrite 0x0384e7 thru 0x039a1f (0x1539 bytes) 000092[0] 1761277298.883192 mapwrite 0x64bb0f thru 0x64bc2f (0x121 bytes) 000093[1] 1761277298.897342 write 0x96b01d thru 0x96bf93 (0xf77 bytes) 000094[0] 1761277298.903951 trunc from 0x96bf94 to 0x50c5e5 (0x45f9ae bytes) 000096[1] 1761277298.916954 read 0x0bdfe0 thru 0x0bee00 (0xe21 bytes) 000098[1] 1761277298.925343 write 0x72652b thru 0x727f91 (0x1a67 bytes) 000099[0] 1761277298.931722 write 0x26808d thru 0x2683c1 (0x335 bytes) 000100[1] 1761277298.939316 mapwrite 0x563c3b thru 0x5648b7 (0xc7d bytes) 000104[0] 1761277298.965758 read 0x3f4f2b thru 0x3f535f (0x435 bytes) 000105[1] 1761277298.970705 mapwrite 0x2e140a thru 0x2e19bf (0x5b6 bytes) 000106[1] 1761277298.989552 mapread 0x50d54d thru 0x50ed3d (0x17f1 bytes) 000107[0] 1761277298.992328 trunc from 0x727f92 to 0x3df1ef (0x348da2 bytes) 000108[1] 1761277299.004015 write 0x33b0b3 thru 0x33cf0e (0x1e5c bytes) 000112[0] 1761277299.020804 write 0x4ce918 thru 0x4cf304 (0x9ed bytes) 000114[0] 1761277299.027955 write 0x7a3948 thru 0x7a3e29 (0x4e2 bytes) 000117[1] 1761277299.037449 mapread 0x3d34e0 thru 0x3d53fc (0x1f1d bytes) 000119[1] 1761277299.048091 mapread 0x48e850 thru 0x48f23a (0x9eb bytes) truncating to largest ever: 0x822a1f 000124[0] 1761277299.066540 trunc from 0x7a3e2a to 0x822a1e (0x7ebf5 bytes) 000125[1] 1761277299.081836 read 0x1b37f4 thru 0x1b54fc (0x1d09 bytes) 000129[1] 1761277299.097078 mapread 0x21e50e thru 0x21fb0d (0x1600 bytes) 000130[1] 1761277299.101377 trunc from 0x822a1f to 0x20381c (0x61f202 bytes) 000133[0] 1761277299.111318 read 0x0cf10c thru 0x0cfc7e (0xb73 bytes) 000134[0] 1761277299.119053 mapwrite 0x46fccc thru 0x471415 (0x174a bytes) 000135[0] 1761277299.142773 mapread 0x059896 thru 0x05a2d8 (0xa43 bytes) 000137[1] 1761277299.145686 write 0x58b981 thru 0x58ced5 (0x1555 bytes) 000140[0] 1761277299.159797 mapread 0x514613 thru 0x514f85 (0x973 bytes) 000141[0] 1761277299.168851 mapread 0x522ed3 thru 0x524501 (0x162f bytes) 000145[0] 1761277299.182658 mapread 0x2070b6 thru 0x208054 (0xf9f bytes) truncating to largest ever: 0x8c5832 000146[0] 1761277299.188223 trunc from 0x58ced6 to 0x8c5831 (0x33895c bytes) 000147[0] 1761277299.195990 mapwrite 0x543880 thru 0x543d1e (0x49f bytes) 000157[0] 1761277299.227477 trunc from 0x8c5832 to 0x5d9159 (0x2ec6d8 bytes) 000158[0] 1761277299.232184 mapwrite 0x1e8522 thru 0x1e8f6e (0xa4d bytes) 000159[0] 1761277299.247641 trunc from 0x5d915a to 0x84b91a (0x2727c1 bytes) 000165[1] 1761277299.272356 write 0x7b6d06 thru 0x7b748a (0x785 bytes) 000166[1] 1761277299.280498 read 0x4e0861 thru 0x4e0ed4 (0x674 bytes) 000167[0] 1761277299.285148 mapwrite 0x27c5d5 thru 0x27cab0 (0x4dc bytes) 000169[0] 1761277299.302918 mapread 0x0dd4d5 thru 0x0de25d (0xd89 bytes) 000170[1] 1761277299.309902 write 0x0913b3 thru 0x092fe6 (0x1c34 bytes) 000171[0] 1761277299.317945 read 0x541ac0 thru 0x54384f (0x1d90 bytes) 000173[1] 1761277299.326039 write 0x145b53 thru 0x146111 (0x5bf bytes) 000181[0] 1761277299.346116 write 0x1b949e thru 0x1b958a (0xed bytes) truncating to largest ever: 0x991677 000183[0] 1761277299.354844 trunc from 0x84b91b to 0x991676 (0x145d5c bytes) 000184[1] 1761277299.361799 write 0x7225bd thru 0x722a70 (0x4b4 bytes) 000187[1] 1761277299.376008 mapwrite 0x26196e thru 0x26381e (0x1eb1 bytes) 000189[1] 1761277299.401271 mapread 0x256649 thru 0x2566f5 (0xad bytes) 000193[0] 1761277299.410468 write 0x03c08b thru 0x03ca3e (0x9b4 bytes) 000194[0] 1761277299.418581 trunc from 0x991677 to 0x5826fd (0x40ef79 bytes) 000197[0] 1761277299.433305 mapread 0x332919 thru 0x3332a5 (0x98d bytes) 000198[0] 1761277299.439973 read 0x2aa839 thru 0x2aba99 (0x1261 bytes) 000202[0] 1761277299.450534 write 0x89924a thru 0x89aa37 (0x17ee bytes) 000203[1] 1761277299.452093 mapread 0x89801d thru 0x899011 (0xff5 bytes) 000204[0] 1761277299.464999 mapwrite 0x032d8d thru 0x034be2 (0x1e56 bytes) 000210[1] 1761277299.497657 write 0x6bf38f thru 0x6c09e8 (0x165a bytes) 000211[0] 1761277299.503190 trunc from 0x89aa38 to 0x689842 (0x2111f5 bytes) 000214[0] 1761277299.520386 write 0x03132f thru 0x032028 (0xcfa bytes) 000216[1] 1761277299.525247 write 0x68ddaf thru 0x68f20a (0x145c bytes) 000218[0] 1761277299.532822 trunc from 0x68f20b to 0x35b879 (0x333991 bytes) 000219[1] 1761277299.542956 read 0x0a8479 thru 0x0a8660 (0x1e8 bytes) 000222[1] 1761277299.575702 read 0x347c16 thru 0x3480db (0x4c6 bytes) 000223[0] 1761277299.576012 mapwrite 0x500d4b thru 0x502cfa (0x1fb0 bytes) 000224[0] 1761277299.606275 trunc from 0x502cfb to 0x4859bb (0x7d33f bytes) 000225[1] 1761277299.614804 write 0x266228 thru 0x26721a (0xff3 bytes) 000228[1] 1761277299.668341 read 0x09626e thru 0x096dd9 (0xb6c bytes) 000230[1] 1761277299.675794 write 0x95d297 thru 0x95dc21 (0x98b bytes) 000232[0] 1761277299.685190 read 0x550fe0 thru 0x552586 (0x15a7 bytes) 000235[1] 1761277299.702074 write 0x2f8eb6 thru 0x2f92b5 (0x400 bytes) 000236[0] 1761277299.705829 write 0x837da8 thru 0x838802 (0xa5b bytes) 000237[1] 1761277299.721604 read 0x2eb32e thru 0x2ecca8 (0x197b bytes) 000239[1] 1761277299.728367 read 0x0f7266 thru 0x0f914f (0x1eea bytes) 000244[0] 1761277299.741516 write 0x95dda3 thru 0x95f28e (0x14ec bytes) 000245[1] 1761277299.746347 mapread 0x389dde thru 0x38a42c (0x64f bytes) 000249[1] 1761277299.763744 mapread 0x523f74 thru 0x525dc6 (0x1e53 bytes) 000253[1] 1761277299.774051 mapwrite 0x2f2fd4 thru 0x2f469f (0x16cc bytes) 000256[1] 1761277299.804127 read 0x190a6c thru 0x19114b (0x6e0 bytes) 000257[0] 1761277299.806090 write 0x01cdb3 thru 0x01d74a (0x998 bytes) 000259[1] 1761277299.815884 mapread 0x584fa8 thru 0x585b37 (0xb90 bytes) 000260[0] 1761277299.825990 read 0x4881dd thru 0x489351 (0x1175 bytes) 000261[0] 1761277299.831161 write 0x007ecd thru 0x0086ff (0x833 bytes) 000262[1] 1761277299.833731 write 0x314707 thru 0x315c14 (0x150e bytes) 000263[1] 1761277299.843370 mapread 0x1a2347 thru 0x1a3c7d (0x1937 bytes) 000266[0] 1761277299.852388 mapread 0x77ba65 thru 0x77c06f (0x60b bytes) truncating to largest ever: 0x9b2716 000267[0] 1761277299.860446 trunc from 0x95f28f to 0x9b2715 (0x53487 bytes) 000268[0] 1761277299.870471 read 0x726330 thru 0x727986 (0x1657 bytes) 000271[1] 1761277299.880937 mapwrite 0x41cc22 thru 0x41d5f8 (0x9d7 bytes) 000276[1] 1761277299.914123 read 0x68d753 thru 0x68eca1 (0x154f bytes) 000277[1] 1761277299.918875 mapwrite 0x25c40a thru 0x25cf0f (0xb06 bytes) 000278[1] 1761277299.934050 write 0x4964d9 thru 0x497c68 (0x1790 bytes) 000279[0] 1761277299.937227 read 0x29360a thru 0x294323 (0xd1a bytes) 000280[1] 1761277299.945014 mapwrite 0x0f5086 thru 0x0f5bdc (0xb57 bytes) 000282[1] 1761277299.965256 read 0x89dc80 thru 0x89e9ba (0xd3b bytes) 000283[1] 1761277299.970725 write 0x26d6e2 thru 0x26eaf8 (0x1417 bytes) 000284[0] 1761277299.973350 mapread 0x97d905 thru 0x97e1bc (0x8b8 bytes) 000286[1] 1761277299.984115 read 0x53a541 thru 0x53bf9a (0x1a5a bytes) 000287[0] 1761277299.988733 mapwrite 0x3eaa8b thru 0x3ead5a (0x2d0 bytes) 000290[0] 1761277300.011698 write 0x4d9543 thru 0x4d9603 (0xc1 bytes) 000291[1] 1761277300.016562 mapwrite 0x7b6c68 thru 0x7b78cf (0xc68 bytes) 000292[1] 1761277300.036237 trunc from 0x9b2716 to 0x756e9a (0x25b87b bytes) 000293[0] 1761277300.043164 mapwrite 0x1af4ef thru 0x1af724 (0x236 bytes) 000295[1] 1761277300.063479 write 0x3c3d93 thru 0x3c4d0e (0xf7c bytes) 000296[1] 1761277300.071966 trunc from 0x756e9b to 0x8a84c5 (0x15162b bytes) 000297[0] 1761277300.079824 trunc from 0x8a84c6 to 0x814838 (0x93c8d bytes) 000300[0] 1761277300.095041 read 0x0b047e thru 0x0b1153 (0xcd6 bytes) 000306[0] 1761277300.116459 read 0x75ee96 thru 0x75f4ff (0x66a bytes) 000307[0] 1761277300.122038 read 0x38217d thru 0x38373c (0x15c0 bytes) 000312[1] 1761277300.133795 read 0x6f3080 thru 0x6f4d18 (0x1c99 bytes) 000313[1] 1761277300.138639 mapwrite 0x9ce8d2 thru 0x9cf3ff (0xb2e bytes) 000315[0] 1761277300.159507 write 0x3f1684 thru 0x3f3044 (0x19c1 bytes) 000316[1] 1761277300.166608 read 0x2669cb thru 0x2687a8 (0x1dde bytes) 000318[1] 1761277300.175051 write 0x5338e4 thru 0x5343b2 (0xacf bytes) 000319[1] 1761277300.180932 trunc from 0x9cf400 to 0x50b2f9 (0x4c4106 bytes) 000320[0] 1761277300.187870 mapread 0x32fe24 thru 0x3300f2 (0x2cf bytes) 000325[1] 1761277300.204370 trunc from 0x50b2fa to 0x7def60 (0x2d3c67 bytes) 000328[1] 1761277300.222694 mapread 0x754f70 thru 0x7550bb (0x14c bytes) 000330[0] 1761277300.226257 mapread 0x6b37c5 thru 0x6b42e7 (0xb23 bytes) 000333[0] 1761277300.237732 mapread 0x77d496 thru 0x77def5 (0xa60 bytes) 000334[1] 1761277300.247924 trunc from 0x7def61 to 0x014040 (0x7caf20 bytes) 000335[0] 1761277300.256801 read 0x00f6ad thru 0x00f7cd (0x121 bytes) 000336[0] 1761277300.266058 read 0x00bd66 thru 0x00c094 (0x32f bytes) 000338[0] 1761277300.274416 mapwrite 0x9beb3c thru 0x9bf6bc (0xb81 bytes) 000340[1] 1761277300.389004 read 0x27300b thru 0x274365 (0x135b bytes) 000341[0] 1761277300.398499 trunc from 0x9bf6bd to 0x0bd107 (0x9025b5 bytes) 000342[1] 1761277300.409303 write 0x45d1bf thru 0x45d9e6 (0x828 bytes) 000345[1] 1761277300.423026 mapwrite 0x6ff26a thru 0x6ff298 (0x2f bytes) 000347[0] 1761277300.443511 read 0x144249 thru 0x145c09 (0x19c1 bytes) 000348[0] 1761277300.451118 mapwrite 0x06dcd2 thru 0x06f677 (0x19a6 bytes) 000349[0] 1761277300.477587 mapwrite 0x9a2da7 thru 0x9a4362 (0x15bc bytes) 000350[0] 1761277300.505247 mapwrite 0x6423b1 thru 0x64403d (0x1c8d bytes) 000351[0] 1761277300.520986 write 0x2bdb0d thru 0x2bf332 (0x1826 bytes) 000355[1] 1761277300.533544 read 0x227929 thru 0x227b27 (0x1ff bytes) 000357[0] 1761277300.544952 write 0x709baf thru 0x709cb9 (0x10b bytes) 000359[1] 1761277300.550928 mapwrite 0x6a3070 thru 0x6a4fbc (0x1f4d bytes) 000365[0] 1761277300.584449 read 0x119032 thru 0x119832 (0x801 bytes) 000369[1] 1761277300.589521 trunc from 0x9a4363 to 0x4c01b7 (0x4e41ab bytes) 000371[0] 1761277300.600702 write 0x5a4c5a thru 0x5a5270 (0x617 bytes) 000373[0] 1761277300.608930 trunc from 0x5a5271 to 0x1a53dd (0x3ffe93 bytes) 000374[0] 1761277300.618779 read 0x170cb9 thru 0x1719af (0xcf7 bytes) 000375[1] 1761277300.632746 write 0x3fc02d thru 0x3fde4d (0x1e21 bytes) 000376[0] 1761277300.640378 mapread 0x03ffdb thru 0x0401bb (0x1e1 bytes) 000378[0] 1761277300.671364 trunc from 0x3fde4e to 0x373b41 (0x8a30c bytes) 000381[0] 1761277300.685439 mapread 0x1abec2 thru 0x1acda9 (0xee8 bytes) 000382[0] 1761277300.688542 mapread 0x11e265 thru 0x11ebd8 (0x974 bytes) 000383[0] 1761277300.693739 read 0x15c1ae thru 0x15db62 (0x19b5 bytes) 000387[1] 1761277300.732425 mapwrite 0x1aecd7 thru 0x1af4a3 (0x7cd bytes) 000388[1] 1761277300.755541 write 0x1bc139 thru 0x1bd1d9 (0x10a1 bytes) 000392[0] 1761277300.773767 trunc from 0x373b42 to 0x8a6bce (0x53308d bytes) 000393[1] 1761277300.784268 mapread 0x2fa0ea thru 0x2fb27e (0x1195 bytes) 000395[0] 1761277300.792847 read 0x871313 thru 0x872f1c (0x1c0a bytes) 000396[1] 1761277300.794685 mapwrite 0x96fdf9 thru 0x970e1c (0x1024 bytes) 000401[0] 1761277300.833492 read 0x698e40 thru 0x69a3e2 (0x15a3 bytes) 000403[1] 1761277300.843243 read 0x8c2001 thru 0x8c2be0 (0xbe0 bytes) 000405[1] 1761277300.852801 read 0x7b5f84 thru 0x7b6cb1 (0xd2e bytes) 000406[1] 1761277300.858866 read 0x894062 thru 0x895c63 (0x1c02 bytes) 000409[1] 1761277300.865880 read 0x859601 thru 0x85b183 (0x1b83 bytes) 000410[0] 1761277300.868280 mapwrite 0x8f7c5e thru 0x8f819a (0x53d bytes) 000411[1] 1761277300.889257 mapwrite 0x633198 thru 0x63425f (0x10c8 bytes) 000412[1] 1761277300.909102 read 0x91bca5 thru 0x91dab9 (0x1e15 bytes) 000415[1] 1761277300.920960 mapread 0x53799f thru 0x538a54 (0x10b6 bytes) 000416[0] 1761277300.923881 mapwrite 0x0ba678 thru 0x0bae6b (0x7f4 bytes) 000419[1] 1761277300.948484 mapread 0x605cb4 thru 0x60644d (0x79a bytes) 000420[0] 1761277300.956405 mapread 0x40d35c thru 0x40d69c (0x341 bytes) 000421[0] 1761277300.965419 write 0x13cc51 thru 0x13ea85 (0x1e35 bytes) 000425[1] 1761277300.981284 trunc from 0x970e1d to 0x86fb6e (0x1012ae bytes) 000426[1] 1761277300.992673 trunc from 0x86fb6f to 0x02ccf0 (0x842e7e bytes) 000427[0] 1761277300.999843 mapwrite 0x20ee95 thru 0x2102e8 (0x1454 bytes) 000428[0] 1761277301.022334 mapread 0x09f24d thru 0x0a0f09 (0x1cbd bytes) 000430[0] 1761277301.031141 trunc from 0x2102e9 to 0x606a15 (0x3f672d bytes) 000431[0] 1761277301.042182 mapwrite 0x6078da thru 0x609420 (0x1b47 bytes) 000432[1] 1761277301.065147 read 0x243efb thru 0x244683 (0x789 bytes) 000433[1] 1761277301.074112 mapwrite 0x21aa6d thru 0x21c063 (0x15f7 bytes) 000435[0] 1761277301.106392 read 0x286be5 thru 0x288a0d (0x1e29 bytes) 000436[1] 1761277301.114763 read 0x26106a thru 0x2623ea (0x1381 bytes) 000439[0] 1761277301.120848 read 0x5a91aa thru 0x5aaa02 (0x1859 bytes) 000442[0] 1761277301.130110 mapwrite 0x499d09 thru 0x49a363 (0x65b bytes) 000443[1] 1761277301.146566 read 0x1efed8 thru 0x1f1555 (0x167e bytes) 000445[1] 1761277301.153026 mapwrite 0x3675e4 thru 0x368e12 (0x182f bytes) 000446[1] 1761277301.176734 mapread 0x05d499 thru 0x05e5f0 (0x1158 bytes) 000449[1] 1761277301.191678 trunc from 0x609421 to 0x77d648 (0x174228 bytes) 000451[0] 1761277301.201044 write 0x00081a thru 0x000feb (0x7d2 bytes) 000452[0] 1761277301.209416 mapread 0x104558 thru 0x105892 (0x133b bytes) 000453[0] 1761277301.217868 trunc from 0x77d649 to 0x895971 (0x118329 bytes) 000454[1] 1761277301.224937 write 0x50adc9 thru 0x50c1ba (0x13f2 bytes) 000455[1] 1761277301.233691 trunc from 0x895972 to 0x2cca28 (0x5c8f49 bytes) 000457[1] 1761277301.246563 mapread 0x0a4123 thru 0x0a4d3b (0xc19 bytes) 000460[1] 1761277301.255408 trunc from 0x2cca29 to 0x7de008 (0x5115e0 bytes) 000461[1] 1761277301.265518 trunc from 0x7de009 to 0x92f2ac (0x1512a4 bytes) 000462[1] 1761277301.275214 mapread 0x3c084e thru 0x3c1e1c (0x15cf bytes) 000463[0] 1761277301.279582 write 0x5e9b97 thru 0x5eb48b (0x18f5 bytes) 000466[1] 1761277301.295271 trunc from 0x92f2ad to 0x834ed8 (0xfa3d4 bytes) 000470[1] 1761277301.304727 mapwrite 0x20c18b thru 0x20d9c2 (0x1838 bytes) 000475[0] 1761277301.323518 mapwrite 0x146b86 thru 0x147ea1 (0x131c bytes) 000476[0] 1761277301.343971 trunc from 0x834ed9 to 0x12d918 (0x7075c0 bytes) 000477[0] 1761277301.350546 mapwrite 0x035c3c thru 0x037449 (0x180e bytes) 000478[0] 1761277301.364803 write 0x4b3eb1 thru 0x4b4bf5 (0xd45 bytes) 000479[0] 1761277301.365977 read 0x056c5a thru 0x058b04 (0x1eab bytes) 000482[0] 1761277301.376345 write 0x516498 thru 0x5179bd (0x1526 bytes) 000486[0] 1761277301.384564 trunc from 0x5179be to 0x62cf5e (0x1155a1 bytes) 000487[0] 1761277301.389565 trunc from 0x62cf5f to 0x762072 (0x135114 bytes) 000489[0] 1761277301.397913 read 0x6db4ca thru 0x6dd07b (0x1bb2 bytes) 000490[0] 1761277301.399676 mapread 0x2aa214 thru 0x2aa22a (0x17 bytes) 000493[0] 1761277301.409567 mapwrite 0x584a64 thru 0x585970 (0xf0d bytes) 000494[1] 1761277301.422758 mapread 0x2dfaf8 thru 0x2dfc63 (0x16c bytes) 000495[0] 1761277301.430766 write 0x8d516e thru 0x8d6759 (0x15ec bytes) 000497[1] 1761277301.438354 read 0x2d56d0 thru 0x2d590f (0x240 bytes) 000498[0] 1761277301.444472 trunc from 0x8d675a to 0x3d660e (0x50014b bytes) 000499[1] 1761277301.454352 mapwrite 0x1e9905 thru 0x1eb07e (0x177a bytes) 000500[1] 1761277301.472806 mapwrite 0x0581b9 thru 0x059664 (0x14ac bytes) All operations completed A-OK! PASS 16b (5s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 23:41:42 (1761277302) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (2s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 23:41:44 (1761277304) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.113873 s, 73.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0697716 s, 120 MB/s ldlm.namespaces.MGC192.168.204.145@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99e1885c6800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff99e1a10ae000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99e1885c6800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99e1885c6800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.40559 s, 43.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.694862 s, 151 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.821253 s, 128 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.35641 s, 77.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.51261 s, 69.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.10606 s, 94.8 MB/s PASS 16d (11s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 23:41:55 (1761277315) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0926098 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333786 s, 31.4 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 23:41:58 (1761277318) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 23:42:20 (1761277340) TIMEOUT PASS 16g (22s) == sanityn test 16h: mmap read after truncate file ======= 23:42:42 (1761277362) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 23:42:45 (1761277365) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 23:42:47 (1761277367) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.566801 s, 46.2 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.366529 s, 71.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.588512 s, 44.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.4283 s, 61.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.540884 s, 48.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.615723 s, 42.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.71221 s, 36.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.596959 s, 43.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.692979 s, 37.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.800311 s, 32.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.830372 s, 31.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.503682 s, 52.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.565035 s, 46.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.693305 s, 37.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.712637 s, 36.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.786436 s, 33.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.87828 s, 29.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.950032 s, 27.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.978853 s, 26.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (10s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 23:42:57 (1761277377) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x32cce75 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Oct 23 23:43:07 EDT 2025 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu Oct 23 23:43:07 EDT 2025 + fsx 3 done: rc=0 Thu Oct 23 23:43:07 EDT 2025 All operations completed A-OK! + fsx 4 done: rc=0 Thu Oct 23 23:43:07 EDT 2025 + fsx 5 done: rc=0 Thu Oct 23 23:43:07 EDT 2025 PASS 16k (13s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 23:43:10 (1761277390) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 23:43:15 (1761277395) mmap test1: basic mmap operation (PASS, 0.025019s) mmap test2: MAP_PRIVATE not write back (PASS, 0.015028s) mmap test3: concurrent mmap ops on two nodes (PASS, 4.1169s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.0984s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.03635s) mmap test6: check mmap write/read content on two nodes (PASS, 0.040783s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.40192s) mmap test11: multi-thread shared mmap (PASS, 0.35127s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0161s) PASS 18 (13s) == sanityn test 19: test concurrent uncached read races ========================================================================= 23:43:28 (1761277408) oleg445-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (1s) == sanityn test 20: test extra readahead page left in cache ============================================================== 23:43:29 (1761277409) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 23:43:32 (1761277412) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (2s) == sanityn test 23: others should see updated atime while another read============================================================== 23:43:34 (1761277414) now is 1761277415 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 new atime is 1761277476 PASS 23 (63s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 23:44:37 (1761277477) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 3.2M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 6.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 32.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 490.6K 388 490.2K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 402 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 399 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 388 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 502379 388 501991 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118066 402 117664 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117231 399 116832 1% /mnt/lustre2[OST:1] filesystem_summary: 234884 388 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 6144 3763200 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 32768 3736576 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 23:44:39 (1761277479) PASS 24b (2s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 23:44:42 (1761277482) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 23:44:44 (1761277484) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 23:44:45 (1761277485) utime: good mknod times 1761277484 <= 1761277485 <= 1761277485 for /mnt/lustre/f26a utime: good mknod times 1761277484 <= 1761277485 <= 1761277485 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 23:44:48 (1761277488) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 23:44:52 (1761277492) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 23:44:53.000000000 -0400 Birth: 2025-10-23 23:44:53.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 23:44:53.000000000 -0400 Birth: 2025-10-23 23:44:53.000000000 -0400 PASS 26c (3s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 23:44:56 (1761277496) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 23:45:00 (1761277500) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 23:45:03 (1761277503) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg445-server mds-ost sync done. sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.204.145@tcp:/lustre 7542784 39936 7496704 1% /mnt/lustre2 PASS 31b (20s) == sanityn test 31r: open-rename(replace) race =========== 23:45:23 (1761277523) -rw-r--r-- 1 root root 0 Oct 23 23:45 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (5s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 23:45:28 (1761277528) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42998 Terminated tail -f $tfile PASS 31s (3s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 23:45:31 (1761277531) PASS 31t (4s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 23:45:36 (1761277536) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 23:45:38 (1761277538) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 23:45:40 (1761277540) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 23:45:42 (1761277542) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (1s) == sanityn test 34: no lock timeout under IO ============= 23:45:44 (1761277544) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid in FULL state after 0 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid in IDLE state after 0 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid in FULL state after 0 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid in IDLE state after 0 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99e1885c6800.ost_server_uuid in FULL state after 0 sec oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff99e1885c6800.ost_server_uuid in IDLE state after 0 sec PASS 34 (94s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 23:47:18 (1761277638) fail_loc=0x80000317 Race attempt 0 Wait for 49123 49258 for 60 sec... total: 4000 open/close in 8.83 seconds: 453.19 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.204.145@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.204.145@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (67s) == sanityn test 36: handle ESTALE/open-unlink correctly == 23:48:25 (1761277705) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg445-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.654556 s, 80.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.445808 s, 118 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.526814 s, 99.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.422035 s, 124 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.505591 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.44165 s, 36.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.20944 s, 43.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7495680) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.538326 s, 97.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7491584) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.516042 s, 102 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.5203 s, 101 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.452806 s, 116 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7446528) after(7497728) PASS 36 (253s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 23:52:38 (1761277958) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 - create 2893 (time 1761277970.44 total 10.01 last 289.12) - create 4244 (time 1761277980.44 total 20.01 last 135.06) - create 5641 (time 1761277990.44 total 30.01 last 139.69) - create 7046 (time 1761278000.45 total 40.02 last 140.42) - create 8413 (time 1761278010.45 total 50.02 last 136.64) - create 9770 (time 1761278020.46 total 60.03 last 135.61) total: 10000 create in 61.79 seconds: 161.83 ops/second PASS 37 (70s) == sanityn test 39a: file mtime does not change after rename ========================================================== 23:53:48 (1761278028) repeat after cancel_lru_locks PASS 39a (5s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 23:53:53 (1761278033) repeat after cancel_lru_locks PASS 39b (4s) == sanityn test 39c: check truncate mtime update ================================================================================ 23:53:57 (1761278037) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 23:54:01 (1761278041) fail_loc=0x411 fail_loc=0 PASS 39d (3s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 23:54:05 (1761278045) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 23:54:23 (1761278063) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (20s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 23:54:43 (1761278083) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (18s) == sanityn test 40e: pdirops: rename and others ======================================================================== 23:55:02 (1761278102) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 23:55:18 (1761278118) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (10s) == sanityn test 41b: pdirops: create vs create ======================================================================== 23:55:29 (1761278129) open: File exists Conflict PASS 41b (10s) == sanityn test 41c: pdirops: create vs link ======================================================================== 23:55:39 (1761278139) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (9s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 23:55:48 (1761278148) Conflict PASS 41d (9s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 23:55:57 (1761278157) Conflict PASS 41e (9s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 23:56:06 (1761278166) Conflict PASS 41f (10s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 23:56:16 (1761278176) Conflict PASS 41g (10s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 23:56:26 (1761278186) Conflict PASS 41h (10s) == sanityn test 41i: reint_open: create vs create ======== 23:56:37 (1761278197) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (770s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 00:09:26 (1761278966) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (6s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 00:09:32 (1761278972) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 00:09:37 (1761278977) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (6s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 00:09:43 (1761278983) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 00:09:48 (1761278988) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 00:09:53 (1761278993) Conflict PASS 42f (6s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 00:09:59 (1761278999) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 00:10:04 (1761279004) Conflict PASS 42h (6s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 00:10:10 (1761279010) PASS 43a (17s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 00:10:27 (1761279027) Conflict PASS 43b (5s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 00:10:32 (1761279032) Conflict PASS 43c (6s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 00:10:38 (1761279038) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 00:10:43 (1761279043) Conflict PASS 43e (6s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 00:10:49 (1761279049) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 00:10:54 (1761279054) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (6s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 00:11:00 (1761279060) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 00:11:05 (1761279065) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 00:11:06 (1761279066) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists PASS 43j (49s) == sanityn test 43k: unlink vs create ==================== 00:11:55 (1761279115) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (482s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 00:19:57 (1761279597) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Fri Oct 24 00:20:01 EDT 2025 PASS 44a (6s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 00:20:03 (1761279603) open: File exists Conflict PASS 44b (6s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 00:20:09 (1761279609) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (6s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 00:20:15 (1761279615) Conflict PASS 44d (6s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 00:20:21 (1761279621) Conflict PASS 44e (6s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 00:20:27 (1761279627) Conflict PASS 44f (6s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 00:20:33 (1761279633) Conflict PASS 44g (6s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 00:20:39 (1761279639) Conflict PASS 44h (6s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 00:20:45 (1761279645) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (1s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 00:20:46 (1761279646) PASS 45a (32s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 00:21:18 (1761279678) Conflict PASS 45b (5s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 00:21:23 (1761279683) Conflict PASS 45c (6s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 00:21:29 (1761279689) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (6s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 00:21:35 (1761279695) Conflict PASS 45e (5s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 00:21:40 (1761279700) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (6s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 00:21:46 (1761279706) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (5s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 00:21:51 (1761279711) Conflict PASS 45h (5s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 00:21:56 (1761279716) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (1s) == sanityn test 45j: read vs rename ====================== 00:21:57 (1761279717) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (531s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 00:30:48 (1761280248) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (6s) == sanityn test 46b: pdirops: link vs create ======================================================================== 00:30:54 (1761280254) open: File exists Conflict PASS 46b (6s) == sanityn test 46c: pdirops: link vs link ======================================================================== 00:31:00 (1761280260) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (6s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 00:31:06 (1761280266) Conflict PASS 46d (6s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 00:31:12 (1761280272) Conflict PASS 46e (6s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 00:31:18 (1761280278) Conflict PASS 46f (6s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 00:31:25 (1761280285) Conflict PASS 46g (6s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 00:31:31 (1761280291) Conflict PASS 46h (6s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 00:31:37 (1761280297) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (1s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 00:31:38 (1761280298) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (1s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 00:31:39 (1761280299) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (1s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 00:31:40 (1761280300) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (2s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 00:31:42 (1761280302) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (1s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 00:31:43 (1761280303) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (1s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 00:31:44 (1761280304) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (1s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 00:31:45 (1761280305) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (2s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 00:31:47 (1761280307) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0095515 s, 1.1 MB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (7s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 00:31:54 (1761280314) 0+1 records in 0+1 records out 158 bytes copied, 0.00596082 s, 26.5 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3510: kill: (260496) - No such process PASS 51a (5s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 00:31:59 (1761280319) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00664805 s, 154 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224225 s, 46.8 MB/s 3145728 PASS 51b (14s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 00:32:13 (1761280333) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00485996 s, 211 kB/s PASS 51c (7s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 00:32:20 (1761280340) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156429 s, 67.0 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (4s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 00:32:24 (1761280344) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2452 0x994 0x280000400 read lease(1) has applied. PASS 51e (4s) == sanityn test 54: rename locking ======================= 00:32:28 (1761280348) llite.lustre-ffff99e1885c6800.xattr_cache=1 llite.lustre-ffff99e1a10ae000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:29.000000000 -0400 Modify: 2025-10-24 00:32:29.000000000 -0400 Change: 2025-10-24 00:32:29.000000000 -0400 Birth: 2025-10-24 00:32:29.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:34.000000000 -0400 Modify: 2025-10-24 00:32:34.000000000 -0400 Change: 2025-10-24 00:32:34.000000000 -0400 Birth: 2025-10-24 00:32:35.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:40.000000000 -0400 Modify: 2025-10-24 00:32:40.000000000 -0400 Change: 2025-10-24 00:32:40.000000000 -0400 Birth: 2025-10-24 00:32:40.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:40.000000000 -0400 Modify: 2025-10-24 00:32:40.000000000 -0400 Change: 2025-10-24 00:32:40.000000000 -0400 Birth: 2025-10-24 00:32:40.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:45.000000000 -0400 Modify: 2025-10-24 00:32:45.000000000 -0400 Change: 2025-10-24 00:32:45.000000000 -0400 Birth: 2025-10-24 00:32:46.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 11776 Blocks: 23 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:32:45.000000000 -0400 Modify: 2025-10-24 00:32:45.000000000 -0400 Change: 2025-10-24 00:32:45.000000000 -0400 Birth: 2025-10-24 00:32:46.000000000 -0400 PASS 54 (24s) == sanityn test 55a: rename vs unlink target dir ========= 00:32:52 (1761280372) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (8s) == sanityn test 55b: rename vs unlink source dir ========= 00:33:00 (1761280380) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (8s) == sanityn test 55c: rename vs unlink orphan target dir == 00:33:08 (1761280388) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (13s) == sanityn test 55d: rename file vs link ================= 00:33:21 (1761280401) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (10s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 00:33:31 (1761280411) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (1s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 00:33:32 (1761280412) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (13s) == sanityn test 55g: rename: race with trylock =========== 00:33:45 (1761280425) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (14s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 00:34:00 (1761280440) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1761280440 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 2.03464 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (33s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 00:34:32 (1761280472) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1761280474 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 3.39095 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (41s) == sanityn test 60: Verify data_version behaviour ======== 00:35:13 (1761280513) PASS 60 (2s) == sanityn test 70a: cd directory && rm directory ======== 00:35:15 (1761280515) PASS 70a (3s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 00:35:18 (1761280518) PASS 70b (3s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 00:35:22 (1761280522) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00308404 s, 21.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (1s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 00:35:23 (1761280523) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00410859 s, 16.0 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (1s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 00:35:24 (1761280524) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (2s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 00:35:26 (1761280526) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00756362 s, 8.7 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (1s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 00:35:27 (1761280527) llite.lustre-ffff99e1885c6800.xattr_cache=1 llite.lustre-ffff99e1a10ae000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (2s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 00:35:29 (1761280529) llite.lustre-ffff99e1885c6800.xattr_cache=1 llite.lustre-ffff99e1a10ae000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (3s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 00:35:32 (1761280532) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00405524 s, 253 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.004028 s, 254 kB/s 273265: get lock2 273267: get lock1 273267: done 273267: get lock2 273265: done 273265: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 273267: cannot get lock2: Resource deadlock avoided 273267: exit rc=0 273265: done 273265: exit rc=0 PASS 74 (6s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 00:35:38 (1761280538) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0275801 s, 76.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164441 s, 63.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4193: 273848 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (8s) == sanityn test 76: Verify MDT open_files listing ======== 00:35:46 (1761280546) opening files: ........ closing files: ........ PASS 76 (21s) == sanityn test 77a: check FIFO NRS policy =============== 00:36:07 (1761280567) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147786 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372544 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629527 s, 16.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053405 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435543 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707778 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0680627 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841551 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717508 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104249 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0987873 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922033 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962682 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098898 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758402 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894429 s, 11.7 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742245 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223991 s, 46.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228259 s, 45.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236128 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268516 s, 39.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00707396 s, 148 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289728 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00810477 s, 129 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260644 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00802546 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185224 s, 56.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300556 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250443 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0116235 s, 90.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283701 s, 37.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261338 s, 40.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199887 s, 52.5 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (4s) == sanityn test 77b: check CRR-N NRS policy ============== 00:36:11 (1761280571) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.168092 s, 99.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233063 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342143 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852126 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.064992 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733814 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655028 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751755 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818308 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090434 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862553 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106335 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107785 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115107 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126389 s, 8.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124334 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131197 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213987 s, 49.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254622 s, 41.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254372 s, 41.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255133 s, 41.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033073 s, 31.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214723 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211703 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253668 s, 41.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235324 s, 44.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236014 s, 44.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234465 s, 44.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358183 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276793 s, 37.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0395462 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339688 s, 30.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344902 s, 30.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.156988 s, 107 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314086 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914443 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951401 s, 11.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0830772 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833167 s, 12.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100553 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108334 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0972014 s, 10.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835317 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925699 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108594 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0989476 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101411 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125354 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119693 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119659 s, 8.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990878 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180689 s, 58.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147322 s, 71.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274662 s, 38.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104214 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163487 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162459 s, 64.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237892 s, 44.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152461 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227719 s, 46.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218998 s, 47.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233278 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00516876 s, 203 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023955 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026322 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.007641 s, 137 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (4s) == sanityn test 77c: check ORR NRS policy ================ 00:36:15 (1761280575) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.155294 s, 108 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293202 s, 35.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485425 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565399 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.053093 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912379 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986939 s, 10.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865511 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100557 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057191 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724059 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958121 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929729 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0790971 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0817983 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0955842 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933165 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0142782 s, 73.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192551 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0157326 s, 66.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162191 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102365 s, 102 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00484526 s, 216 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188823 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00433834 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180466 s, 58.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221337 s, 47.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00327177 s, 320 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251195 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299035 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00378507 s, 277 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00909519 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129272 s, 81.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.160697 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322604 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403553 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419339 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566384 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0826114 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867182 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797518 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078388 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100842 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0905925 s, 11.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110658 s, 9.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105037 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920352 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102612 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114618 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192999 s, 54.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241588 s, 43.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156019 s, 67.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269587 s, 38.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026169 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234206 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162931 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224166 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288087 s, 36.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0296898 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283279 s, 37.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00367416 s, 285 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246025 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00932079 s, 112 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294196 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00581769 s, 180 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=2 removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.143084 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241128 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472402 s, 22.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553164 s, 19.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0734214 s, 14.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062223 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711808 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756818 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0619841 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599609 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122814 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105648 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133785 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142337 s, 7.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131134 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139815 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140103 s, 7.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260468 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161888 s, 64.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020366 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197331 s, 53.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309343 s, 33.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198057 s, 52.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021539 s, 48.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149117 s, 70.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253192 s, 41.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165484 s, 63.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0271601 s, 38.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258114 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219537 s, 47.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00646831 s, 162 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250418 s, 41.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282969 s, 37.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (6s) == sanityn test 77d: check TRR nrs policy ================ 00:36:21 (1761280581) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.171281 s, 98.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325841 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487952 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739762 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887952 s, 11.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948911 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864213 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819653 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10766 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104894 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107757 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104997 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12685 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147911 s, 7.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137179 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144624 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170283 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221927 s, 47.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040587 s, 258 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00968256 s, 108 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194428 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280606 s, 37.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=9 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00393007 s, 267 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00394751 s, 266 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00416815 s, 252 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00419957 s, 250 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00425345 s, 247 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420926 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327948 s, 32.0 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303379 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01145 s, 91.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106929 s, 98.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556634 s, 188 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145951 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283836 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034181 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313884 s, 33.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0413976 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931551 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752884 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.084572 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0996782 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116659 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112404 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124918 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111406 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979658 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137295 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122205 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134916 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197331 s, 53.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189941 s, 55.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182505 s, 57.5 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206988 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104735 s, 100 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231569 s, 45.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00608793 s, 172 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0276965 s, 37.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00915607 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279287 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192467 s, 54.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261578 s, 40.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152415 s, 68.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00826075 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399507 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168547 s, 62.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.159848 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320537 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0478429 s, 21.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859994 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879866 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0879012 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914639 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100307 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979533 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969529 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0857511 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964151 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847342 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520523 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.09065 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653889 s, 16.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953866 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202488 s, 51.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0198672 s, 52.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214893 s, 48.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018308 s, 57.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245983 s, 42.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232165 s, 45.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278474 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0241227 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246972 s, 42.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135316 s, 77.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219024 s, 47.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299083 s, 35.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00520011 s, 202 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126097 s, 83.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0089351 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00953807 s, 110 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (5s) == sanityn test 77e: check TBF NID nrs policy ============ 00:36:26 (1761280586) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.204.45@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.173721 s, 96.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393634 s, 26.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063071 s, 16.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055802 s, 18.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601326 s, 17.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675103 s, 15.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686803 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055598 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0620605 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0681844 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784088 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100039 s, 10.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100847 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113668 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108026 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119282 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116716 s, 9.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147748 s, 71.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134926 s, 77.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218009 s, 48.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259961 s, 40.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00979221 s, 107 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169598 s, 61.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00455507 s, 230 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00841054 s, 125 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163625 s, 64.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0223938 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256895 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025292 s, 41.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255618 s, 41.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00727737 s, 144 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239181 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00629883 s, 166 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=2 removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.147266 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035727 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039161 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411722 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329909 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500498 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733029 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0662287 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645238 s, 16.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105746 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100531 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10766 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901971 s, 11.6 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104177 s, 10.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106582 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114495 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115923 s, 9.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365049 s, 28.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360503 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385387 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381433 s, 27.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353822 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0403395 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405607 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.042617 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527974 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506363 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0419248 s, 25.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524112 s, 20.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422691 s, 24.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471401 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540502 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594535 s, 17.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.14209 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202428 s, 51.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0332888 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020256 s, 51.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121633 s, 8.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112892 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118859 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142244 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140963 s, 7.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144844 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133569 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133422 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16848 s, 6.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182248 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1784 s, 5.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176537 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179664 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255198 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033071 s, 31.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224101 s, 46.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277827 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212115 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263946 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229786 s, 45.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028664 s, 36.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02746 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273167 s, 38.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0315299 s, 33.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295588 s, 35.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336245 s, 31.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352699 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351353 s, 29.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390468 s, 26.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.135184 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033509 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377347 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752971 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780356 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642768 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697469 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967104 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675379 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672305 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114849 s, 9.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106657 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950518 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106157 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114214 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986569 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113574 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173582 s, 60.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00470466 s, 223 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214736 s, 48.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216258 s, 48.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278361 s, 37.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0339084 s, 30.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275158 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299707 s, 35.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310145 s, 33.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263805 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0204295 s, 51.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215628 s, 48.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493384 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143822 s, 72.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019971 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184901 s, 56.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (8s) == sanityn test 77f: check TBF JobID nrs policy ========== 00:36:34 (1761280594) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.170017 s, 98.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0350066 s, 30.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060206 s, 17.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600709 s, 17.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650859 s, 16.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922059 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714365 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0982561 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0993173 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0977591 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693877 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801089 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771364 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100716 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0931344 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107554 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10332 s, 10.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173125 s, 60.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207897 s, 50.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146223 s, 71.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184993 s, 56.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256985 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166272 s, 63.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236018 s, 44.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236052 s, 44.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242994 s, 43.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240143 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245849 s, 42.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00557415 s, 188 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00376988 s, 278 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400365 s, 262 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0040033 s, 262 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00330471 s, 317 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.131957 s, 127 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360088 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06118 s, 17.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871628 s, 12.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761794 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067805 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0733318 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917856 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079996 s, 13.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865235 s, 12.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872425 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936028 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994934 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118543 s, 8.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104506 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114179 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104832 s, 10.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140729 s, 74.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176459 s, 59.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161758 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172957 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189194 s, 55.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158503 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0055554 s, 189 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0140854 s, 74.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282578 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0055902 s, 188 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144177 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191122 s, 54.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135797 s, 77.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0262791 s, 39.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105292 s, 99.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380435 s, 27.6 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.161626 s, 104 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423373 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561106 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519646 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049045 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443893 s, 23.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631951 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0548487 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487984 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0631021 s, 16.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126383 s, 8.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=1 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134149 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160997 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143006 s, 7.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167439 s, 6.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13976 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188968 s, 5.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172569 s, 60.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267489 s, 39.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236908 s, 44.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166389 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00449028 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00593044 s, 177 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00630668 s, 166 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00648004 s, 162 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.022425 s, 46.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00624871 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794652 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.029868 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00509692 s, 206 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00531885 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187354 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00353463 s, 297 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.138042 s, 122 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046087 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049895 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725338 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788854 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698184 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0900102 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908971 s, 11.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10342 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994953 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120355 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124865 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108432 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112492 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116784 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115084 s, 9.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125271 s, 8.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff99e1a10ae000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff99e1a10ae000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00783791 s, 134 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151543 s, 69.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216275 s, 48.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167093 s, 62.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175089 s, 59.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209817 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187232 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00889173 s, 118 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00930475 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207899 s, 50.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122165 s, 85.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01664 s, 63.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00536622 s, 195 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00751337 s, 140 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027512 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879749 s, 119 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (9s) == sanityn test 77g: Change TBF type directly ============ 00:36:43 (1761280603) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (4s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 00:36:47 (1761280607) oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg445-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg445-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg445-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg445-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 PASS 77h (4s) == sanityn test 77i: Change rank of TBF rule ============= 00:36:51 (1761280611) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (8s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 00:36:59 (1761280619) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.148558 s, 113 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268632 s, 39.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372916 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668828 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089194 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940876 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116346 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114802 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108624 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979007 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116263 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108373 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111558 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126355 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10602 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109529 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115819 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176417 s, 59.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263749 s, 39.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144183 s, 72.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318719 s, 32.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251355 s, 41.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327061 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250749 s, 41.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224515 s, 46.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0298991 s, 35.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243001 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308462 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247636 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255021 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321366 s, 32.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327247 s, 32.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286911 s, 36.5 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97684 s, 21.1 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4358 s, 5.4 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (49s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 00:37:48 (1761280668) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22556 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753298 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0755545 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694579 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745168 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842923 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055116 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756678 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696738 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15676 s, 6.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156727 s, 6.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173933 s, 6.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152749 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148061 s, 7.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154336 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178225 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153909 s, 6.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227774 s, 46.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0232355 s, 45.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246873 s, 42.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185337 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259673 s, 40.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263988 s, 39.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274023 s, 38.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250428 s, 41.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314697 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019973 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00756306 s, 139 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152395 s, 68.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144002 s, 72.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0117268 s, 89.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03688 s, 28.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124065 s, 84.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8343 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9442 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22436 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835256 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0750803 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103776 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929599 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104322 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120755 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119313 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125536 s, 8.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180928 s, 5.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180872 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196713 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192884 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205841 s, 5.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173573 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212711 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193901 s, 5.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173422 s, 60.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208495 s, 50.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216355 s, 48.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205423 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01668 s, 62.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277385 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219785 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0210656 s, 49.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026626 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00723095 s, 145 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345531 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285916 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146408 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00519512 s, 202 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00535888 s, 196 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00210351 s, 498 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8353 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9511 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (133s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 00:40:01 (1761280801) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22631 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045224 s, 23.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658378 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738967 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822031 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10786 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0737344 s, 14.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889634 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944783 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153219 s, 6.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153236 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125431 s, 8.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15042 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158432 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163009 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160479 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185848 s, 5.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188124 s, 55.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163532 s, 64.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191499 s, 54.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220032 s, 47.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0221438 s, 47.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184721 s, 56.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234084 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199459 s, 52.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00901343 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247757 s, 42.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00418696 s, 250 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277628 s, 37.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316494 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00453945 s, 231 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275031 s, 38.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00722582 s, 145 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8406 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9583 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.22681 s, 13.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0464904 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374058 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520439 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0454761 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0425722 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060865 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582137 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054009 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151775 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159014 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177671 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178432 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197986 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209264 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199868 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22831 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150235 s, 69.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164834 s, 63.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178961 s, 58.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189928 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023097 s, 45.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205083 s, 51.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00385703 s, 272 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255315 s, 41.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212275 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172658 s, 60.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00439503 s, 239 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243557 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233995 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00420452 s, 249 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00329549 s, 318 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00400044 s, 262 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8339 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9506 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (133s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 00:42:15 (1761280935) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260434 s, 64.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421343 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881868 s, 11.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0994169 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106411 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920445 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114507 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106418 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112867 s, 9.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958659 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128323 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106098 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110642 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109354 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117203 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111227 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123845 s, 8.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229709 s, 45.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243646 s, 43.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189367 s, 55.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0224876 s, 46.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015519 s, 67.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226352 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305384 s, 34.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188961 s, 55.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185618 s, 56.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375255 s, 27.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199248 s, 52.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144995 s, 72.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357992 s, 29.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0111296 s, 94.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00910913 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206237 s, 50.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99657 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73865 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.127713 s, 131 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0480375 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0825217 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113774 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991289 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135074 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105268 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110211 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126426 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14042 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126788 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133172 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119467 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115621 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12864 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115282 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135709 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432656 s, 242 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188793 s, 55.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189058 s, 55.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226876 s, 46.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018241 s, 57.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0208421 s, 50.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0138683 s, 75.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216335 s, 48.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00883337 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258753 s, 40.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218774 s, 47.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177254 s, 59.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229188 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193564 s, 54.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231053 s, 45.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133769 s, 78.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93375 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73929 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.123862 s, 135 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243313 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307818 s, 34.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831316 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067043 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753108 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815421 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0779524 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08526 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871989 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998375 s, 10.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104959 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834288 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841073 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106691 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867131 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980724 s, 10.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159482 s, 65.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182573 s, 57.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161152 s, 65.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0122128 s, 85.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0169585 s, 61.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195643 s, 53.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104408 s, 100 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233122 s, 45.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207091 s, 50.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00651798 s, 161 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00363106 s, 289 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217761 s, 48.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193621 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240023 s, 43.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267288 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00222448 s, 471 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94692 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.88968 s, 21.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.129232 s, 130 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273757 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266977 s, 39.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897442 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801293 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969823 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11597 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131793 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128306 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133255 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126555 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108071 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145768 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148024 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118441 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161408 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149638 s, 7.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101432 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0165709 s, 63.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0188661 s, 55.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184103 s, 57.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187112 s, 56.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00452301 s, 232 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0349707 s, 30.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288581 s, 36.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360776 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367577 s, 28.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108571 s, 96.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00784424 s, 134 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0430534 s, 24.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393669 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114503 s, 91.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00288874 s, 363 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77849 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73789 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.91879 s, 21.3 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73696 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.843516 s, 19.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380317 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835465 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796782 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836343 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831509 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101973 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103873 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935525 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155729 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186341 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221974 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191286 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219869 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207532 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213537 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237406 s, 4.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143528 s, 73.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0135656 s, 77.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186457 s, 56.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167292 s, 62.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00625035 s, 168 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0102206 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161292 s, 65.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184266 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0151945 s, 69.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245103 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0205673 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211148 s, 49.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214571 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00387577 s, 271 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00548019 s, 191 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00204748 s, 512 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6577 s, 5.3 MB/s Write runtime is 20 s, speed is 5.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.4421 s, 5.4 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.24605 s, 68.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509863 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471664 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762498 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102331 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105946 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925189 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115503 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0932607 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850191 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868066 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109738 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114148 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111313 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102444 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134472 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116265 s, 9.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119311 s, 87.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176619 s, 59.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216916 s, 48.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154082 s, 68.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240567 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212273 s, 49.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00834521 s, 126 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00463514 s, 226 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235696 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172131 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00451043 s, 232 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303275 s, 34.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023303 s, 45.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00892537 s, 117 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00409517 s, 256 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325249 s, 32.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.94309 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73824 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136037 s, 123 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304033 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0398087 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0390898 s, 26.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807134 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0571744 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0872796 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944286 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960734 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109663 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0898561 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0912995 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103598 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109359 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908658 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116051 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112421 s, 9.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185036 s, 56.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199872 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0180972 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206399 s, 50.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.023429 s, 44.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220557 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283888 s, 36.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489724 s, 21.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260726 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490889 s, 21.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0305558 s, 34.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553995 s, 18.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316967 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317778 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342631 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355347 s, 29.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77948 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73963 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77684 s, 10.7 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73738 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (356s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 00:48:10 (1761281290) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.263171 s, 63.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273695 s, 38.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057987 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721715 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703872 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104464 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0911283 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0908859 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893104 s, 11.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113463 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929249 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104996 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101857 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117927 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961499 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114133 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107295 s, 9.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0163567 s, 64.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0202762 s, 51.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274732 s, 38.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172181 s, 60.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191533 s, 54.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325198 s, 32.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132993 s, 78.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329389 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345228 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319389 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179688 s, 58.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388382 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0168922 s, 62.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0213153 s, 49.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194563 s, 53.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0156839 s, 66.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93611 s, 21.2 MB/s Write runtime is 5 s, speed is 20.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73687 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (37s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 00:48:47 (1761281327) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.204.45@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.204.45@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.136958 s, 122 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289231 s, 36.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922625 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0985591 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914107 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880157 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810332 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103618 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966022 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106735 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915392 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897761 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103849 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086312 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860354 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0894022 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751487 s, 14.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184327 s, 56.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174623 s, 60.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203371 s, 51.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0192835 s, 54.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209872 s, 50.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191501 s, 54.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0108383 s, 96.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226601 s, 46.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00644298 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00765832 s, 137 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0195205 s, 53.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247844 s, 42.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.014564 s, 72.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00641071 s, 164 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317206 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125463 s, 83.6 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.0404 s, 20.8 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73902 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (40s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 00:49:27 (1761281367) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.204.45@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.145417 s, 115 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0477968 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556942 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.055715 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437089 s, 24.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0739024 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698512 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565308 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0847209 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066943 s, 15.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123733 s, 8.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134069 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1128 s, 9.3 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157991 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148201 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144308 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144679 s, 7.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0257423 s, 40.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031052 s, 33.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289273 s, 36.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027801 s, 37.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234705 s, 44.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306684 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323231 s, 32.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277115 s, 37.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0265103 s, 39.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0275673 s, 38.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345751 s, 30.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148296 s, 70.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226283 s, 46.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173486 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0129432 s, 81.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00868837 s, 121 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.99781 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.9741 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (31s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 00:49:59 (1761281399) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.126224 s, 133 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0269169 s, 39.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702335 s, 14.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763278 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607048 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07149 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660242 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591743 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076881 s, 13.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744329 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098015 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13016 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106305 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11515 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140678 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121736 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161314 s, 6.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00966256 s, 109 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0148231 s, 70.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199877 s, 52.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193157 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167555 s, 62.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252743 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152598 s, 68.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193626 s, 54.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234439 s, 44.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019373 s, 54.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100264 s, 105 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187812 s, 55.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00411749 s, 255 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196925 s, 53.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00821724 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00404974 s, 259 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.82796 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73934 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.93765 s, 21.2 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74067 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (80s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 00:51:19 (1761281479) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.852087 s, 19.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321 s, 32.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510952 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0261119 s, 40.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0544332 s, 19.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621925 s, 16.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758896 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714241 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715849 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158027 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198228 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.215611 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216985 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21872 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221784 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227585 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243638 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176969 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191071 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132878 s, 78.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164529 s, 63.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182301 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252218 s, 41.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02335 s, 44.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291087 s, 36.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302372 s, 34.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.025631 s, 40.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0306302 s, 34.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358981 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474368 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00622422 s, 168 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113998 s, 92.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0343367 s, 30.5 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7952 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9539 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (66s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 00:52:25 (1761281545) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.260728 s, 64.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521178 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435143 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794881 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814905 s, 12.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766054 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0760342 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649524 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922323 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751871 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881685 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130644 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154049 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145012 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164518 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133757 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135162 s, 7.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02093 s, 50.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0190025 s, 55.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234989 s, 44.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028545 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272861 s, 38.4 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024491 s, 42.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00790461 s, 133 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0193771 s, 54.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0344711 s, 30.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0182893 s, 57.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297118 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317329 s, 33.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016446 s, 63.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309555 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0252509 s, 41.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119841 s, 87.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.98364 s, 21.0 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73566 s, 10.8 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg445-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.121045 s, 139 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031171 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0289958 s, 36.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0922292 s, 11.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10522 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0836744 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0918364 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119189 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114732 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112026 s, 9.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11188 s, 9.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981813 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107061 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111556 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126741 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106433 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108998 s, 9.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00429577 s, 244 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150667 s, 69.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021218 s, 49.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0152751 s, 68.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173061 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0189131 s, 55.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0203728 s, 51.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220054 s, 47.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197645 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566614 s, 185 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222506 s, 47.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012868 s, 81.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240723 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474724 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00513023 s, 204 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00219208 s, 478 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.84139 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73832 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.77443 s, 10.7 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.73834 s, 10.8 MB/s Read runtime is 10 s, speed is 10.000000 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (119s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 00:54:24 (1761281664) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (3s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 00:54:27 (1761281667) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg445-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.48151 s, 301 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg445-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0155 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (51s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 00:55:18 (1761281718) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.151902 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324613 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668966 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617463 s, 17.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554291 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676855 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116008 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990663 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143455 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150485 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142161 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133879 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142685 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155964 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151663 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189369 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163726 s, 6.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019884 s, 52.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0217879 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00598512 s, 175 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024244 s, 43.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297247 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267435 s, 39.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399022 s, 26.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297078 s, 35.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295979 s, 35.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300719 s, 34.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0303583 s, 34.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316385 s, 33.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00456602 s, 230 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884318 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323003 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364579 s, 28.8 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg445-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg445-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.00679 s, 20.9 MB/s Write runtime is 6 s, speed is 16.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97359 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.153492 s, 109 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.021023 s, 49.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690805 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075214 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820737 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952739 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909403 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105578 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846952 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592599 s, 17.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109418 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109673 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941385 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721788 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107278 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888865 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0870385 s, 12.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020165 s, 52.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207706 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0174719 s, 60.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187693 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00413491 s, 254 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251743 s, 41.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134747 s, 77.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263475 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239787 s, 43.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243348 s, 43.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00884119 s, 119 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.02699 s, 38.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00711557 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0282272 s, 37.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00493142 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00399394 s, 263 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg445-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg445-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.02432 s, 20.9 MB/s Write runtime is 7 s, speed is 14.285714 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.97466 s, 21.1 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (66s) == sanityn test 77o: Changing rank should not panic ====== 00:56:24 (1761281784) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (4s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 00:56:28 (1761281788) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (41s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 00:57:09 (1761281829) mds.MDS.mdt.nrs_policies=tbf oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg445-client: oleg445-server: ssh exited with exit code 36 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg445-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg445-client: oleg445-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (13s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 00:57:22 (1761281842) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 309859 kill md_thread xargs PASS 77r (40s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 00:58:02 (1761281882) oleg445-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_orr_quantum=1: No such device oleg445-server: error: set_param: setting 'ost/OSS/*/nrs_orr_quantum'='1': No such device pdsh@oleg445-client: oleg445-server: ssh exited with exit code 19 ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_policies=fifo PASS 78 (3s) == sanityn test 79: xattr: intent error ================== 00:58:05 (1761281885) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272506460 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 00:58:06.000000000 -0400 Modify: 2025-10-24 00:58:06.000000000 -0400 Change: 2025-10-24 00:58:06.000000000 -0400 Birth: 2025-10-24 00:58:07.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (13s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 00:58:18 (1761281898) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (1s) == sanityn test 80b: Accessing directory during migration ========================================================== 00:58:19 (1761281899) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (1s) == sanityn test 81a: rename and stat under striped directory ========================================================== 00:58:20 (1761281900) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (1s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 00:58:22 (1761281902) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (1s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 00:58:23 (1761281903) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (1s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 00:58:24 (1761281904) total: 2500 open/close in 5.81 seconds: 430.30 ops/second snapshot_time 1761281929.664754120 secs.nsecs start_time 1761281906.053998770 secs.nsecs elapsed_time 23.610755350 secs.nsecs open 2505 samples [usecs] 91 2236 920972 405744450 close 2504 samples [usecs] 67 986 285420 44707224 mknod 2501 samples [usecs] 183 2128 806780 318227730 unlink 2500 samples [usecs] 291 35305 1709586 6133794388 mkdir 2 samples [usecs] 660 904 1564 1252816 rmdir 2 samples [usecs] 2019 3495 5514 16291386 rename 2500 samples [usecs] 266 14401 3536885 7302036043 getattr 3760 samples [usecs] 7 3510 95072 22721350 parallel_rename_file 2500 samples [usecs] 266 14401 3536885 7302036043 crossdir_rename 2500 samples [usecs] 266 14401 3536885 7302036043 PASS 81d (26s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 00:58:50 (1761281930) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7551 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7551 PASS 82 (2s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 00:58:52 (1761281932) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (1s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 00:58:53 (1761281933) fail_loc=0x8000060b PASS 84 (8s) == sanityn test 85: Lustre API root cache race =========== 00:59:01 (1761281941) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1761281941 Finishing test test1 at 1761281942 PASS 85 (2s) == sanityn test 90: open/create and unlink striped directory ========================================================== 00:59:04 (1761281944) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (2s) == sanityn test 91: chmod and unlink striped directory === 00:59:05 (1761281945) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (1s) == sanityn test 92: create remote directory under orphan directory ========================================================== 00:59:06 (1761281946) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (1s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 00:59:07 (1761281947) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5780 0x1694 0x280000400 0 5806 0x16ae 0x240000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5807 0x16af 0x240000400 1 5781 0x1695 0x280000400 rc1=0 and rc2=0 PASS 93 (8s) == sanityn test 94: signal vs CP callback race =========== 00:59:15 (1761281955) osc.lustre-OST0000-osc-ffff99e1885c6800.idle_timeout=0 osc.lustre-OST0000-osc-ffff99e1a10ae000.idle_timeout=0 osc.lustre-OST0001-osc-ffff99e1885c6800.idle_timeout=0 osc.lustre-OST0001-osc-ffff99e1a10ae000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.040198 s, 204 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 3.09039 s, 1.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5988: 327928 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0128054 s, 320 kB/s osc.lustre-OST0000-osc-ffff99e1885c6800.idle_timeout=debug osc.lustre-OST0000-osc-ffff99e1a10ae000.idle_timeout=debug osc.lustre-OST0001-osc-ffff99e1885c6800.idle_timeout=debug osc.lustre-OST0001-osc-ffff99e1a10ae000.idle_timeout=debug PASS 94 (11s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 00:59:26 (1761281966) llite.lustre-ffff99e1885c6800.fast_read=0 llite.lustre-ffff99e1a10ae000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0219054 s, 95.7 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff99e1885c6800.fast_read=1 llite.lustre-ffff99e1a10ae000.fast_read=1 PASS 95a (14s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 00:59:40 (1761281980) llite.lustre-ffff99e1885c6800.fast_read=0 llite.lustre-ffff99e1a10ae000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00179023 s, 6.9 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff99e1885c6800.fast_read=1 llite.lustre-ffff99e1a10ae000.fast_read=1 PASS 95b (8s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 00:59:48 (1761281988) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (1s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 00:59:50 (1761281990) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00134521 s, 3.0 MB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (2s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 00:59:52 (1761281992) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.066887 s, 31.4 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (2s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 00:59:54 (1761281994) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.063664 s, 32.9 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (2s) == sanityn test 100e: DoM: read on open and file size ==== 00:59:56 (1761281996) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00190495 s, 6.3 MB/s PASS 100e (3s) == sanityn test 101a: Discard DoM data on unlink ========= 00:59:59 (1761281999) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000441845 s, 9.3 MB/s PASS 101a (2s) == sanityn test 101b: Discard DoM data on rename ========= 01:00:01 (1761282001) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000582632 s, 7.0 MB/s PASS 101b (3s) == sanityn test 101c: Discard DoM data on close-unlink === 01:00:04 (1761282004) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00066374 s, 6.2 MB/s PASS 101c (3s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 01:00:08 (1761282008) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1761282009 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1761282012 Finishing test test23 at 1761282012 PASS 103 (7s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 01:00:16 (1761282016) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (1s) == sanityn test 105: Glimpse and lock cancel race ======== 01:00:17 (1761282017) fail_loc=0x416 PASS 105 (17s) == sanityn test 106a: Verify the btime via statx() ======= 01:00:34 (1761282034) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (1s) == sanityn test 106b: Glimpse RPCs test for statx ======== 01:00:36 (1761282036) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00774031 s, 132 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508995 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:00:36.000000000 -0400 Modify: 2025-10-24 01:00:36.000000000 -0400 Change: 2025-10-24 01:00:36.000000000 -0400 Birth: 2025-10-24 01:00:37.000000000 -0400 /mnt/lustre/f106b.sanityn 144115205272508995 -rw-r--r-- 1 0 0 1761282037 1761282036 1761282036 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272508995 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:00:36.000000000 -0400 Modify: 2025-10-24 01:00:36.000000000 -0400 Change: 2025-10-24 01:00:36.000000000 -0400 Birth: 2025-10-24 01:00:37.000000000 -0400 1761282036 1024 1 PASS 106b (2s) == sanityn test 106c: Verify statx attributes mask ======= 01:00:38 (1761282038) PASS 106c (2s) == sanityn test 107a: Basic grouplock conflict =========== 01:00:40 (1761282040) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.093366 s, 112 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5787 0x169b 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 First grouplock blocks second one PASS 107a (5s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 01:00:45 (1761282045) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.113908 s, 92.1 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5816 0x16b8 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7551 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (8s) == sanityn test 108a: lseek: parallel updates ============ 01:00:53 (1761282053) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00212015 s, 4.7 MB/s llite.lustre-ffff99e1885c6800.hybrid_io=0 llite.lustre-ffff99e1a10ae000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.34415 s, 1.9 MB/s PASS 108a (7s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 01:01:00 (1761282060) 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Starting race between client mount instances (50 iterations): Iteration 1 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 3 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 4 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 5 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 6 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 7 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 8 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 9 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 10 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 11 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 12 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 13 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 14 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 15 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 16 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 17 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 18 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 19 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 20 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 21 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 22 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 23 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 24 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 25 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 26 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 27 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 28 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 29 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 30 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 31 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 32 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 33 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 34 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 35 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 36 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 37 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 38 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 39 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 40 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 41 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 42 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 43 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 44 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 45 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 46 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 47 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 48 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 49 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Iteration 50 Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 192.168.204.145@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.204.145@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre2 PASS 109 (515s) == sanityn test 110: do not grant another lock on resend ========================================================== 01:09:35 (1761282575) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205306056707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2025-10-24 01:09:36.000000000 -0400 Modify: 2025-10-24 01:09:36.000000000 -0400 Change: 2025-10-24 01:09:36.000000000 -0400 Birth: 2025-10-24 01:09:36.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg445-client.virtnet oleg445-client.virtnet PASS 110 (65s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 01:10:40 (1761282640) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (1s) == sanityn test 112: update max-inherit in default LMV === 01:10:41 (1761282641) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (1s) == sanityn test 113: check servers of specified fs ======= 01:10:42 (1761282642) lustre-OST0000-osc-ffff99e18a923800 active. lustre-OST0001-osc-ffff99e18a923800 active. lustre-OST0000-osc-ffff99e18a923800 active. lustre-OST0001-osc-ffff99e18a923800 active. MGC192.168.204.145@tcp active. PASS 113 (2s) == sanityn test 114: implicit default LMV inherit ======== 01:10:44 (1761282644) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (1s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 01:10:45 (1761282645) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (1s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 01:10:46 (1761282646) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (1s) == sanityn test 117a: TCU: Init and enable Trash Can on MDTs ========================================================== 01:10:47 (1761282647) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_enable=0 PASS 117a (3s) == sanityn test 117b: Move regular file and empty dir into trash can dir ========================================================== 01:10:50 (1761282650) SKIP: sanityn test_117b ldiskfs only test SKIP 117b (1s) == sanityn test 117c: Move deleted tree with multiple levels into trash ========================================================== 01:10:52 (1761282652) SKIP: sanityn test_117c ldiskfs only test SKIP 117c (1s) == sanityn test 117d: Per-User Trash can Type testing ==== 01:10:53 (1761282653) SKIP: sanityn test_117d ldiskfs only test SKIP 117d (1s) == sanityn test 117e: Undeleted dir in trash should keep its original xattrs ========================================================== 01:10:54 (1761282654) SKIP: sanityn test_117e ldiskfs only test SKIP 117e (1s) == sanityn test 117f: Uncache the dentry under the trash dir ========================================================== 01:10:55 (1761282655) SKIP: sanityn test_117f ldiskfs only test SKIP 117f (1s) == sanityn test 117g: Access .Trash for a non-striped directory ========================================================== 01:10:56 (1761282656) SKIP: sanityn test_117g ldiskfs only test SKIP 117g (1s) == sanityn test 117h: Permission testing of various operations for files in Trash Can ========================================================== 01:10:57 (1761282657) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain Trash Can Type: plain Trash list: ===> /mnt/lustre/.lustre/trash/MDT0000: 0x200000403:0x5:0x0 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0: tdir1 tfile3 /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/tdir1: <=== touch: setting times of '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/f117h.sanityn.crfail': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.crfail': Operation not permitted rmdir: failed to remove '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.crfail': No such file or directory mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/f117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/f117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/f117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/d117h.sanityn.mvfail': Operation not permitted mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/d117h.sanityn.mvfail': No data available mv: cannot move '/mnt/lustre/d117h.sanityn/d117h.sanityn.mvfail' to '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0/d117h.sanityn.mvfail': No data available chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3': Bad file descriptor truncate: cannot truncate '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3' to length 4096: Bad file descriptor chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chattr: Bad file descriptor while setting flags on /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1 chmod: changing permissions of '/mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1': Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tfile3: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor setfattr: /mnt/lustre/.lustre/trash/MDT0000/0x200000403:0x5:0x0//tdir1: Bad file descriptor open: Bad file descriptor open: Bad file descriptor read: Operation not permitted mmap: Operation not permitted open: Is a directory open: Is a directory mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid PASS 117h (4s) == sanityn test 117i: Forbid to create a file with name .Trash ========================================================== 01:11:02 (1761282662) mkdir: cannot create directory '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument touch: setting times of '/mnt/lustre/d117i.sanityn/.Trash': No such file or directory open(O_RDWR|O_CREAT): Invalid argument mcreate: cannot create `/mnt/lustre/d117i.sanityn/.Trash' with mode 0100644: Invalid argument ln: failed to create hard link '/mnt/lustre/d117i.sanityn/.Trash' => '/mnt/lustre/d117i.sanityn/f117i.sanityn': Invalid argument ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/f117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' ln: failed to create symbolic link '/mnt/lustre/d117i.sanityn/.Trash': Invalid argument mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/d117i.sanityn/.Trash' mv: cannot move '/mnt/lustre/d117i.sanityn/d117i.sanityn' to a subdirectory of itself, '/mnt/lustre/.Trash' PASS 117i (2s) == sanityn test 117j: Move back file from Trash Can ====== 01:11:04 (1761282664) SKIP: sanityn test_117j ldiskfs only test SKIP 117j (1s) == sanityn test 117k: Unrm file from Trash Can =========== 01:11:05 (1761282665) SKIP: sanityn test_117k ldiskfs only test SKIP 117k (1s) == sanityn test 117l: Clean up files in Trash Can via command 'lfs trash clean' ========================================================== 01:11:06 (1761282666) SKIP: sanityn test_117l ldiskfs only test SKIP 117l (1s) == sanityn test 117m: Empty Trash Can via the command 'lfs trash clean' ========================================================== 01:11:07 (1761282667) SKIP: sanityn test_117m ldiskfs only test SKIP 117m (1s) == sanityn test 117n: Update LinkEA accordingly when moving file into Trash Can ========================================================== 01:11:08 (1761282668) SKIP: sanityn test_117n ldiskfs only test SKIP 117n (1s) == sanityn test 117o: Set/Clear NOTRASH flag on file ===== 01:11:09 (1761282669) SKIP: sanityn test_117o ldiskfs only test SKIP 117o (1s) == sanityn test 117p: Trash Can handling for remote dir/file ========================================================== 01:11:10 (1761282670) SKIP: sanityn test_117p needs >= 2 MDTs SKIP 117p (1s) == sanityn test 117q: Access Trash Can via '.Trash' for striped directory ========================================================== 01:11:11 (1761282671) SKIP: sanityn test_117q needs >= 2 MDTs SKIP 117q (1s) == sanityn test 117r: Verify the deleting a striped directory on Trash Can ========================================================== 01:11:12 (1761282672) SKIP: sanityn test_117r ldiskfs only test SKIP 117r (1s) == sanityn test 117s: Create trash shard if it does not exist ========================================================== 01:11:13 (1761282673) SKIP: sanityn test_117s ldiskfs only test SKIP 117s (1s) == sanityn test 117t: UID/GID/ProjID for files in Trash Can ========================================================== 01:11:14 (1761282674) SKIP: sanityn test_117t ldiskfs only test SKIP 117t (1s) == sanityn test 117u: statfs() call should not account the space in Trash Can ========================================================== 01:11:15 (1761282675) SKIP: sanityn test_117u ldiskfs only test SKIP 117u (1s) == sanityn test 117ub: statfs() account the inode usage in Trash Can ========================================================== 01:11:17 (1761282677) SKIP: sanityn test_117ub ldiskfs only test SKIP 117ub (1s) == sanityn test 200: service remains healthy while able to process request ========================================================== 01:11:18 (1761282678) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=19 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0893 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.084 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.099 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.103 s, 0.2 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 20.094 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0933 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0982 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0909 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.0949 s, 0.2 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0911 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1116 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1428 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1297 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1042 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1016 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1449 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1142 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 40.141 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1627 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1897 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2267 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1978 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2201 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1885 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.259 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.2507 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.1981 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2759 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2952 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2936 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3035 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2723 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.2791 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3158 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3095 s, 0.1 kB/s4096 bytes (4.1 kB, 4.0 KiB) copied, 80.3013 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.344 s, 0.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 100.353 s, 0.0 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg445-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff99e18a923800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff99e18a923800.ost_server_uuid in FULL state after 0 sec PASS 200 (107s) cleanup: ====================================================== == sanityn test complete, duration 5846 sec ============== 01:13:05 (1761282785) === sanityn: start cleanup 01:13:06 (1761282786) === Stopping clients: oleg445-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg445-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 01:13:51 (1761282831) ===